Re: [PR] Docs and Website XTable Rename [incubator-xtable]
kywe665 commented on PR #352: URL: https://github.com/apache/incubator-xtable/pull/352#issuecomment-1985214325 Alright, the logo vote thread has passed, and I believe all current feedback has been resolved. Super appreciate the thorough reviews and the patience while I missed a few easy things along the way! Thank you! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@xtable.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Docs and Website XTable Rename [incubator-xtable]
kywe665 commented on code in PR #352: URL: https://github.com/apache/incubator-xtable/pull/352#discussion_r1517351712 ## README.md: ## @@ -1,73 +1,81 @@ -# OneTable +# Apache XTable™ (Incubating) [](https://dev.azure.com/onetable-io/onetable-io/_build/latest?definitionId=1=main) -OneTable is an omni-directional converter for table formats that facilitates interoperability across data processing systems -and query engines. -Currently, OneTable supports widely adopted open-source table formats such as Apache Hudi, Apache Iceberg, and Delta Lake. +Apache XTable™ (Incubating) is a cross-table converter for table formats that facilitates omni-directional interoperability across +data processing systems and query engines. Currently, Apache XTable™ supports widely adopted open-source table formats such as +Apache Hudi, Apache Iceberg, and Delta Lake. -OneTable simplifies data lake operations by leveraging a common model for table representation. -This allows users to write data in one format while still benefiting from integrations and features available in other -formats. -For instance, OneTable enables existing Hudi users to seamlessly work with Databricks's Photon Engine or query Iceberg -tables with Snowflake. -Creating transformations from one format to another is straightforward and only requires the implementation of a few interfaces, -which we believe will facilitate the expansion of supported source and target formats in the future. +Apache XTable™ simplifies data lake operations by leveraging a common model for table representation. This allows users to write +data in one format while still benefiting from integrations and features available in other formats. For instance, +Apache XTable™ enables existing Hudi users to seamlessly work with Databricks's Photon Engine or query Iceberg tables with +Snowflake. Creating transformations from one format to another is straightforward and only requires the implementation +of a few interfaces, which we believe will facilitate the expansion of supported source and target formats in the +future. # Building the project and running tests. -1. Use Java11 for building the project. If you are using some other java version, you can use [jenv](https://github.com/jenv/jenv) to use multiple java versions locally. + Review Comment: Good idea to keep it clean, I believe I removed them all -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@xtable.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Docs and Website XTable Rename [incubator-xtable]
kywe665 commented on code in PR #352: URL: https://github.com/apache/incubator-xtable/pull/352#discussion_r1517325150 ## website/docs/starrocks.md: ## @@ -9,15 +9,15 @@ import TabItem from '@theme/TabItem'; # Querying from StarRocks StarRocks allows you to query table formats like Hudi, Delta and Iceberg tables using our [external catalog](https://docs.starrocks.io/docs/data_source/catalog/catalog_overview/) feature. -Users do not need additional configurations to work with OneTable synced tables. +Users do not need additional configurations to work with XTable synced tables. Review Comment: This is fixed now -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@xtable.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Docs and Website XTable Rename [incubator-xtable]
kywe665 commented on code in PR #352: URL: https://github.com/apache/incubator-xtable/pull/352#discussion_r1517324807 ## website/static/images/xtable-white-svg.svg: ## @@ -0,0 +1,72 @@ +http://www.w3.org/2000/svg;> Review Comment: the image was unused and I removed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@xtable.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Docs and Website XTable Rename [incubator-xtable]
kywe665 commented on code in PR #352: URL: https://github.com/apache/incubator-xtable/pull/352#discussion_r1517324417 ## website/static/index.html: ## @@ -228,22 +212,29 @@ What are the future pl keyboard_arrow_down -Current contributors across Onehouse, Microsoft, Google, and others are planning to incubate OneTable into the Apache Software Foundation. Stay tuned for more updates. +Current contributors across Onehouse, Microsoft, Google, and others are helping to incubate Apache XTable™ into the Apache Software Foundation. Stay tuned for more updates. - + https://onetable.dev/docs/setup/; class="footer-link-three">Docs -https://github.com/onetable-io/onetable; class="footer-link-three">GitHub -https://www.linkedin.com/company/onetable-oss/; class="footer-link-three">LinkedIn -https://twitter.com/OnetableOSS; class="footer-link-three l">Twitter +https://github.com/apache/incubator-xtable; class="footer-link-three">GitHub +https://www.linkedin.com/company/apache-xtable/; class="footer-link-three">LinkedIn +https://twitter.com/apachextable; class="footer-link-three l">Twitter - + + + +https://incubator.apache.org/; target="_blank"> Review Comment: Ok I got the incubator logo referenced from external, thanks for the tip -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@xtable.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Create .asf.yaml file [incubator-xtable]
ashvina commented on code in PR #372: URL: https://github.com/apache/incubator-xtable/pull/372#discussion_r1517235883 ## .asf.yaml: ## @@ -0,0 +1,43 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +github: + description: "Apache XTable (incubating) is a cross-table converter for lakehouse table formats that facilitates interoperability across data processing systems and query engines." + homepage: https://xtable.apache.org/ + labels: +- apache-iceberg +- apache-hudi +- delta-lake + features: +wiki: false +issues: true +projects: true +discussions: true + enabled_merge_buttons: +# "squash and merge" replaces committer with noreply@github, and we don't want that +# See https://lists.apache.org/thread/vxxpt1x316kjryb4dptsbs95p66d9xrv +squash: false +# We prefer linear history, so creating merge commits is disabled in UI +merge: false +rebase: true + protected_branches: +main: + required_pull_request_reviews: +required_approving_review_count: 1 +notifications: +commits: commits@xtable.apache.org +issues: commits@xtable.apache.org Review Comment: Maybe dev list would be a better place for PRs and Issues notifications than commits? I'm not strongly opinionated on this. I just want to confirm if commits-list is deliberate. ## .asf.yaml: ## @@ -0,0 +1,43 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +github: + description: "Apache XTable (incubating) is a cross-table converter for lakehouse table formats that facilitates interoperability across data processing systems and query engines." + homepage: https://xtable.apache.org/ + labels: +- apache-iceberg +- apache-hudi +- delta-lake + features: +wiki: false +issues: true +projects: true +discussions: true + enabled_merge_buttons: +# "squash and merge" replaces committer with noreply@github, and we don't want that +# See https://lists.apache.org/thread/vxxpt1x316kjryb4dptsbs95p66d9xrv +squash: false Review Comment: When squash and merge is disabled, should all the PRs squash the changes before merging? This is the recommendation in the discussion link above for Apache Drill. ## .asf.yaml: ## @@ -0,0 +1,43 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +github: + description: "Apache XTable (incubating) is a cross-table converter for lakehouse table formats that facilitates interoperability across data processing systems and query engines." + homepage: https://xtable.apache.org/ + labels: +- apache-iceberg +- apache-hudi +- delta-lake + features: +wiki: false Review Comment: A wiki could be a helpful
Re: [PR] Docs and Website XTable Rename [incubator-xtable]
kywe665 commented on code in PR #352: URL: https://github.com/apache/incubator-xtable/pull/352#discussion_r1517235632 ## website/static/index.html: ## @@ -228,22 +212,29 @@ What are the future pl keyboard_arrow_down -Current contributors across Onehouse, Microsoft, Google, and others are planning to incubate OneTable into the Apache Software Foundation. Stay tuned for more updates. +Current contributors across Onehouse, Microsoft, Google, and others are planning to incubate XTable into the Apache Software Foundation. Stay tuned for more updates. Review Comment: I actually re-read this and now deleted this section from the Q as it's not as relevant anymore -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@xtable.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Docs and Website XTable Rename [incubator-xtable]
kywe665 commented on code in PR #352: URL: https://github.com/apache/incubator-xtable/pull/352#discussion_r1517235368 ## website/static/images/xtable-svg.svg: ## @@ -0,0 +1,70 @@ +http://www.w3.org/2000/svg;> Review Comment: thank you, I have removed all unused images now -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@xtable.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Support] Request: Ability to Invoke Scala Code for Operations in oneTable [incubator-xtable]
the-other-tim-brown commented on issue #353: URL: https://github.com/apache/incubator-xtable/issues/353#issuecomment-1985052827 This ticket is fine. There is no NullPointerException thrown here. If you inspect the stacktrace you'll see a `ClassNotFoundException`. This means that there is an issue with the classpath. Which version of spark does the Glue job run with? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@xtable.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Support] Request: Ability to Invoke Scala Code for Operations in oneTable [incubator-xtable]
soumilshah1995 commented on issue #353: URL: https://github.com/apache/incubator-xtable/issues/353#issuecomment-1984799106 For Glue Scala should I create separate Ticket or this thread is good ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@xtable.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Move github actions under .github folder [incubator-xtable]
jcamachor merged PR #373: URL: https://github.com/apache/incubator-xtable/pull/373 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@xtable.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(incubator-xtable) branch main updated: Move github actions under .github folder (#373)
This is an automated email from the ASF dual-hosted git repository.
jcamacho pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-xtable.git
The following commit(s) were added to refs/heads/main by this push:
new 6684440 Move github actions under .github folder (#373)
6684440 is described below
commit 6684440a1f5dac06f82a0b9347eb859c50ad5310
Author: Jesús Camacho Rodríguez
AuthorDate: Thu Mar 7 13:41:37 2024 -0800
Move github actions under .github folder (#373)
---
{.azure-pipelines => .github}/workflows/azure_ci.js| 0
{.azure-pipelines => .github}/workflows/azure_ci_check.yml | 0
{.azure-pipelines => .github}/workflows/scheduled_workflow.yml | 2 +-
3 files changed, 1 insertion(+), 1 deletion(-)
diff --git a/.azure-pipelines/workflows/azure_ci.js
b/.github/workflows/azure_ci.js
similarity index 100%
rename from .azure-pipelines/workflows/azure_ci.js
rename to .github/workflows/azure_ci.js
diff --git a/.azure-pipelines/workflows/azure_ci_check.yml
b/.github/workflows/azure_ci_check.yml
similarity index 100%
rename from .azure-pipelines/workflows/azure_ci_check.yml
rename to .github/workflows/azure_ci_check.yml
diff --git a/.azure-pipelines/workflows/scheduled_workflow.yml
b/.github/workflows/scheduled_workflow.yml
similarity index 97%
rename from .azure-pipelines/workflows/scheduled_workflow.yml
rename to .github/workflows/scheduled_workflow.yml
index 1d4b81c..3c774f1 100644
--- a/.azure-pipelines/workflows/scheduled_workflow.yml
+++ b/.github/workflows/scheduled_workflow.yml
@@ -46,7 +46,7 @@ jobs:
script: |
// Cron schedule may not be reliable so giving buffer time to
avoid missing recent PRs
const since = new Date(new Date().getTime() - (900 *
1000)).toISOString();
-const query = `repo:${context.repo.owner}/${context.repo.repo}
type:pr state:open base:master updated:>=${since}`;
+const query = `repo:${context.repo.owner}/${context.repo.repo}
type:pr state:open base:main updated:>=${since}`;
const openPrs = await
github.paginate(github.rest.search.issuesAndPullRequests, {
q: query,
sort: 'updated',
Re: [PR] Move github actions under .github folder [incubator-xtable]
xtable-bot commented on PR #373: URL: https://github.com/apache/incubator-xtable/pull/373#issuecomment-1984549404 ## CI report: * d258d4a73b4eeedc9452aa22f3389a6135500082 Azure: [SUCCESS](https://dev.azure.com/apache-xtable-ci-org/bb368b1b-7955-479d-8cbf-689ff9aa23eb/_build/results?buildId=857) * 05e83f02f113f774f5bd2116678c702420bb1ede UNKNOWN Bot commands @xtable-bot supports the following commands: - `@xtable-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@xtable.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Move github actions under .github folder [incubator-xtable]
jcamachor commented on PR #373: URL: https://github.com/apache/incubator-xtable/pull/373#issuecomment-1984545524 @ashvina , yes, that's right. In fact, these files correspond to GitHub actions that control the status check in GitHub, while the CI itself is controlled by Azure Pipelines workflow files. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@xtable.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Move github actions under .github folder [incubator-xtable]
ashvina commented on PR #373: URL: https://github.com/apache/incubator-xtable/pull/373#issuecomment-1984535181 Thank you @jcamachor I have a small confusion. GitHub searches the .github/workflows directory in your repository for workflow files. Does that mean that the workflow files which are getting moved to .github folder in this repo would also be executed in addition to xtable-ci? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@xtable.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Move github actions under .github folder [incubator-xtable]
xtable-bot commented on PR #373: URL: https://github.com/apache/incubator-xtable/pull/373#issuecomment-1984088227 ## CI report: * d258d4a73b4eeedc9452aa22f3389a6135500082 Azure: [SUCCESS](https://dev.azure.com/apache-xtable-ci-org/bb368b1b-7955-479d-8cbf-689ff9aa23eb/_build/results?buildId=857) Bot commands @xtable-bot supports the following commands: - `@xtable-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@xtable.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Create .asf.yaml file [incubator-xtable]
jcamachor commented on PR #372: URL: https://github.com/apache/incubator-xtable/pull/372#issuecomment-1984050923 @zabetak , would you mind taking a look? Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@xtable.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Move github actions under .github folder [incubator-xtable]
jcamachor commented on PR #373: URL: https://github.com/apache/incubator-xtable/pull/373#issuecomment-1984049389 @ashvina , could you take a look since you reviewed #368? Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@xtable.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Move github actions under .github folder [incubator-xtable]
xtable-bot commented on PR #373: URL: https://github.com/apache/incubator-xtable/pull/373#issuecomment-1984048620 ## CI report: * d258d4a73b4eeedc9452aa22f3389a6135500082 Azure: [PENDING](https://dev.azure.com/apache-xtable-ci-org/bb368b1b-7955-479d-8cbf-689ff9aa23eb/_build/results?buildId=857) Bot commands @xtable-bot supports the following commands: - `@xtable-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@xtable.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Move github actions under .github folder [incubator-xtable]
xtable-bot commented on PR #373: URL: https://github.com/apache/incubator-xtable/pull/373#issuecomment-1984046525 ## CI report: * d258d4a73b4eeedc9452aa22f3389a6135500082 UNKNOWN Bot commands @xtable-bot supports the following commands: - `@xtable-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@xtable.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Move github actions under .github folder [incubator-xtable]
jcamachor opened a new pull request, #373: URL: https://github.com/apache/incubator-xtable/pull/373 ## What is the purpose of the pull request This is an addendum to #368. #368 created some action description file for status check on the Azure CI. Those files should be under .github. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@xtable.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Create .asf.yaml file [incubator-xtable]
xtable-bot commented on PR #372: URL: https://github.com/apache/incubator-xtable/pull/372#issuecomment-1984039352 ## CI report: * 23fe2bf1a165385023268aa64392369f7d334e50 UNKNOWN Bot commands @xtable-bot supports the following commands: - `@xtable-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@xtable.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Create .asf.yaml file [incubator-xtable]
jcamachor opened a new pull request, #372: URL: https://github.com/apache/incubator-xtable/pull/372 ## What is the purpose of the pull request This PR creates the .asf.yaml file for the project. More information about the supported options can be found here: https://github.com/apache/infrastructure-asfyaml/blob/main/README.md The current options basically preserve the behavior that was enabled before the transfer to the ASF organization. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@xtable.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Docs and Website XTable Rename [incubator-xtable]
ashvina commented on code in PR #352: URL: https://github.com/apache/incubator-xtable/pull/352#discussion_r1516512251 ## website/static/images/xtable-write-anywhere.png: ## Review Comment: This policy link is exactly what I needed, thank you! I also agree with your opinion regarding solving it incrementally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@xtable.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Support] Request: Ability to Invoke Scala Code for Operations in oneTable [incubator-xtable]
soumilshah1995 commented on issue #353: URL: https://github.com/apache/incubator-xtable/issues/353#issuecomment-1983479883 I think as sagar was saying there is NULL pointer exception hmm -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@xtable.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [Support] Request: Ability to Invoke Scala Code for Operations in oneTable [incubator-xtable]
soumilshah1995 commented on issue #353:
URL:
https://github.com/apache/incubator-xtable/issues/353#issuecomment-1983449032
Hey @the-other-tim-brown I started some work with glue and onetable on
delta streamer
here is what I am doing and I know 99% its jar issue
# Step 1: Upload jar to S3
```
s3://XX/jar/hudi-spark3.3-bundle_2.12-0.14.0.jar
s3://XX/jar/hudi-utilities-slim-bundle_2.12-0.14.0.jar
s3://XXjar/jcommander-1.78.jar
s3://XX/jar/hudi-extensions-0.1.0-SNAPSHOT-bundled.jar
s3://XX/jar/hudi-java-client-0.14.0.jar
```
# Step 2: Upload Sample dataset inside test folder
Link. ;
https://drive.google.com/drive/folders/1BwNEK649hErbsWcYLZhqCWnaXFX3mIsg?usp=share_link
# Stop 3: Create Glue job with Delta Streamer and onetable
```
import com.amazonaws.services.glue.GlueContext
import com.amazonaws.services.glue.MappingSpec
import com.amazonaws.services.glue.errors.CallSite
import com.amazonaws.services.glue.util.GlueArgParser
import com.amazonaws.services.glue.util.Job
import com.amazonaws.services.glue.util.JsonOptions
import org.apache.spark.SparkContext
import scala.collection.JavaConverters._
import org.apache.spark.sql.SparkSession
import org.apache.spark.api.java.JavaSparkContext
import org.apache.hudi.utilities.streamer.HoodieStreamer
import org.apache.hudi.utilities.streamer.SchedulerConfGenerator
import org.apache.hudi.utilities.UtilHelpers
import com.beust.jcommander.JCommander;
import com.beust.jcommander.Parameter;
object GlueApp {
def main(sysArgs: Array[String]) {
val args = GlueArgParser.getResolvedOptions(sysArgs,
Seq("JOB_NAME").toArray)
val BUCKET = "apply-systems-qa"
var config = Array(
"--source-class", "org.apache.hudi.utilities.sources.ParquetDFSSource",
"--source-ordering-field", "replicadmstimestamp",
s"--target-base-path", s"s3://$BUCKET/testcases/",
"--target-table", "invoice",
"--table-type" , "COPY_ON_WRITE",
"--hoodie-conf",
"hoodie.datasource.write.keygenerator.class=org.apache.hudi.keygen.SimpleKeyGenerator",
"--hoodie-conf", "hoodie.datasource.write.recordkey.field=invoiceid",
"--hoodie-conf",
"hoodie.datasource.write.partitionpath.field=destinationstate",
s"--hoodie-conf",
s"hoodie.streamer.source.dfs.root=s3://$BUCKET/test/",
"--hoodie-conf",
"hoodie.datasource.write.precombine.field=replicadmstimestamp"
)
val cfg = HoodieStreamer.getConfig(config)
val additionalSparkConfigs =
SchedulerConfGenerator.getSparkSchedulingConfigs(cfg)
val jssc = UtilHelpers.buildSparkContext("delta-streamer-test", "jes",
additionalSparkConfigs)
val spark = jssc.sc
val glueContext: GlueContext = new GlueContext(spark)
Job.init(args("JOB_NAME"), glueContext, args.asJava)
try {
new HoodieStreamer(cfg, jssc).sync();
} finally {
jssc.stop();
}
Job.commiimport com
.amazonaws.services.glue.GlueContext
import com.amazonaws.services.glue.MappingSpec
import com.amazonaws.services.glue.errors.CallSite
import com.amazonaws.services.glue.util.GlueArgParser
import com.amazonaws.services.glue.util.Job
import com.amazonaws.services.glue.util.JsonOptions
import org.apache.spark.SparkContext
import scala.collection.JavaConverters._
import org.apache.spark.sql.SparkSession
import org.apache.spark.api.java.JavaSparkContext
import org.apache.hudi.utilities.streamer.HoodieStreamer
import org.apache.hudi.utilities.streamer.SchedulerConfGenerator
import org.apache.hudi.utilities.UtilHelpers
import com.beust.jcommander.JCommander;
import com.beust.jcommander.Parameter;
object GlueApp {
def main(sysArgs: Array[String]) {
val args = GlueArgParser.getResolvedOptions(sysArgs,
Seq("JOB_NAME").toArray)
val BUCKET = "XX"
var config = Array(
"--source-class",
"org.apache.hudi.utilities.sources.ParquetDFSSource",
"--source-ordering-field", "replicadmstimestamp",
s"--target-base-path", s"s3://$BUCKET/testcases/",
"--target-table", "invoice",
"--table-type", "COPY_ON_WRITE",
"--sync-tool-classes", "io.onetable.hudi.sync.OneTableSyncTool",
"--enable-sync",
"--hoodie-conf",
"hoodie.datasource.write.keygenerator.class=org.apache.hudi.keygen.SimpleKeyGenerator",
"--hoodie-conf",
"hoodie.datasource.write.recordkey.field=invoiceid",
"--hoodie-conf",
"hoodie.datasource.write.partitionpath.field=destinationstate",
s"--hoodie-conf",
s"hoodie.streamer.source.dfs.root=s3://$BUCKET/test/",
"--hoodie-conf",
Re: [PR] Docs and Website XTable Rename [incubator-xtable]
zabetak commented on code in PR #352: URL: https://github.com/apache/incubator-xtable/pull/352#discussion_r1515759372 ## website/static/images/xtable-write-anywhere.png: ## Review Comment: I suppose the policy that applies here is: https://www.apache.org/foundation/marks/linking#productsupport But we don't have to solve everything as part of this PR and can do that incrementally. This PR focuses on rename and logo so let's keep the material and changes focused on this aspects. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@xtable.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(incubator-xtable) branch main updated (ebcb9b5 -> 7bd832d)
This is an automated email from the ASF dual-hosted git repository. zabetak pushed a change to branch main in repository https://gitbox.apache.org/repos/asf/incubator-xtable.git from ebcb9b5 Reenable CI using Azure Pipelines (#368) add 7bd832d Upgrade spotless-maven-plugin from 2.27.2 to 2.43.0 (#371) No new revisions were added by this update. Summary of changes: core/src/main/java/io/onetable/delta/DeltaSourceClient.java | 1 + core/src/test/java/io/onetable/delta/TestDeltaSync.java | 1 + pom.xml | 4 ++-- 3 files changed, 4 insertions(+), 2 deletions(-)
Re: [PR] Upgrade spotless-maven-plugin from 2.27.2 to 2.43.0 [incubator-xtable]
zabetak merged PR #371: URL: https://github.com/apache/incubator-xtable/pull/371 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@xtable.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
