Re: [PR] [HUDI-7776] Simplify HoodieStorage instance fetching [hudi]
yihua commented on PR #11292: URL: https://github.com/apache/hudi/pull/11292#issuecomment-2130752937 Azure CI manually triggered: https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=24050 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7790] Revert changes in DFSPathSelector and UtilHelpers.readConfig [hudi]
yihua commented on PR #11294: URL: https://github.com/apache/hudi/pull/11294#issuecomment-2130752865 Azure CI manually triggered: https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=24051 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7794] Bump org.apache.hive:hive-service from 2.3.1 to 2.3.4 [hudi]
yihua commented on PR #11295: URL: https://github.com/apache/hudi/pull/11295#issuecomment-2130752731 Azure CI manually triggered: https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=24053 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7794] Bump org.apache.hive:hive-service from 2.3.1 to 2.3.4 [hudi]
yihua commented on PR #11298: URL: https://github.com/apache/hudi/pull/11298#issuecomment-2130752331 Azure CI manually triggered: https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=24052 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7790][DNM][Testing] Revert changes in DFSPathSelector and UtilHelpers.readConfig [hudi]
yihua closed pull request #11300: [HUDI-7790][DNM][Testing] Revert changes in DFSPathSelector and UtilHelpers.readConfig URL: https://github.com/apache/hudi/pull/11300 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [HUDI-7790][DNM][Testing] Revert changes in DFSPathSelector and UtilHelpers.readConfig [hudi]
yihua opened a new pull request, #11300: URL: https://github.com/apache/hudi/pull/11300 ### Change Logs As above ### Impact Testing https://github.com/apache/hudi/pull/11294 only (Azure CI could not be triggered for the PR) ### Risk level none ### Documentation Update none ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Change Logs and Impact were stated clearly - [ ] Adequate tests were added if applicable - [ ] CI passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(hudi) branch dependabot/maven/org.apache.hive-hive-service-2.3.3 updated (c4838982b79 -> e7e4b9e3ddf)
This is an automated email from the ASF dual-hosted git repository. yihua pushed a change to branch dependabot/maven/org.apache.hive-hive-service-2.3.3 in repository https://gitbox.apache.org/repos/asf/hudi.git from c4838982b79 Update pom.xml add e7e4b9e3ddf Update pom.xml No new revisions were added by this update. Summary of changes: pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
(hudi) branch dependabot/maven/org.apache.hive-hive-service-2.3.3 updated (da65366d649 -> c4838982b79)
This is an automated email from the ASF dual-hosted git repository. yihua pushed a change to branch dependabot/maven/org.apache.hive-hive-service-2.3.3 in repository https://gitbox.apache.org/repos/asf/hudi.git from da65366d649 Bump org.apache.hive:hive-service from 2.3.1 to 2.3.3 add c4838982b79 Update pom.xml No new revisions were added by this update. Summary of changes: packaging/hudi-flink-bundle/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Re: [PR] [HUDI-7790] Revert changes in DFSPathSelector and UtilHelpers.readConfig [hudi]
yihua commented on PR #11294: URL: https://github.com/apache/hudi/pull/11294#issuecomment-2130746129 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(hudi) branch master updated: [HUDI-7790] Revert changes in DFSPathSelector and UtilHelpers.readConfig (#11293)
This is an automated email from the ASF dual-hosted git repository.
yihua pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/master by this push:
new eba95af7ee6 [HUDI-7790] Revert changes in DFSPathSelector and
UtilHelpers.readConfig (#11293)
eba95af7ee6 is described below
commit eba95af7ee6e5b8228a5e59e63d92bc32a618e7c
Author: Y Ethan Guo
AuthorDate: Fri May 24 21:15:22 2024 -0700
[HUDI-7790] Revert changes in DFSPathSelector and UtilHelpers.readConfig
(#11293)
---
.../org/apache/hudi/cli/commands/SparkMain.java| 3 +-
.../hudi/integ/testsuite/HoodieTestSuiteJob.java | 3 +-
.../SparkDataSourceContinuousIngestTool.java | 3 +-
.../helpers/DFSTestSuitePathSelector.java | 41
.../apache/hudi/utilities/HDFSParquetImporter.java | 3 +-
.../hudi/utilities/HoodieDataTableValidator.java | 3 +-
.../hudi/utilities/HoodieDropPartitionsTool.java | 4 +-
.../org/apache/hudi/utilities/HoodieIndexer.java | 4 +-
.../utilities/HoodieMetadataTableValidator.java| 2 +-
.../apache/hudi/utilities/HoodieRepairTool.java| 2 +-
.../org/apache/hudi/utilities/TableSizeStats.java | 3 +-
.../org/apache/hudi/utilities/UtilHelpers.java | 10 ++--
.../multitable/HoodieMultiTableServicesMain.java | 3 +-
.../utilities/sources/helpers/DFSPathSelector.java | 54 ++
.../sources/helpers/DatePartitionPathSelector.java | 46 +++---
.../streamer/HoodieMultiTableStreamer.java | 5 +-
.../hudi/utilities/streamer/HoodieStreamer.java| 5 +-
.../helpers/TestDFSPathSelectorCommonMethods.java | 19
.../helpers/TestDatePartitionPathSelector.java | 5 +-
19 files changed, 102 insertions(+), 116 deletions(-)
diff --git a/hudi-cli/src/main/java/org/apache/hudi/cli/commands/SparkMain.java
b/hudi-cli/src/main/java/org/apache/hudi/cli/commands/SparkMain.java
index 0a43233db8a..bb63a051d9b 100644
--- a/hudi-cli/src/main/java/org/apache/hudi/cli/commands/SparkMain.java
+++ b/hudi-cli/src/main/java/org/apache/hudi/cli/commands/SparkMain.java
@@ -44,7 +44,6 @@ import org.apache.hudi.hadoop.fs.HadoopFSUtils;
import org.apache.hudi.index.HoodieIndex;
import org.apache.hudi.keygen.constant.KeyGeneratorType;
import org.apache.hudi.storage.HoodieStorageUtils;
-import org.apache.hudi.storage.StoragePath;
import org.apache.hudi.table.HoodieSparkTable;
import
org.apache.hudi.table.action.compact.strategy.UnBoundedCompactionStrategy;
import org.apache.hudi.table.marker.WriteMarkersFactory;
@@ -483,7 +482,7 @@ public class SparkMain {
String payloadClassName, String
enableHiveSync, String propsFilePath, List configs) throws IOException {
TypedProperties properties = propsFilePath == null ?
buildProperties(configs)
-: readConfig(jsc.hadoopConfiguration(), new
StoragePath(propsFilePath), configs).getProps(true);
+: readConfig(jsc.hadoopConfiguration(), new Path(propsFilePath),
configs).getProps(true);
properties.setProperty(HoodieBootstrapConfig.BASE_PATH.key(), sourcePath);
diff --git
a/hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/HoodieTestSuiteJob.java
b/hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/HoodieTestSuiteJob.java
index 8813129d748..70910357d7d 100644
---
a/hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/HoodieTestSuiteJob.java
+++
b/hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/HoodieTestSuiteJob.java
@@ -44,7 +44,6 @@ import org.apache.hudi.integ.testsuite.reader.DeltaInputType;
import org.apache.hudi.integ.testsuite.writer.DeltaOutputMode;
import org.apache.hudi.keygen.BuiltinKeyGenerator;
import org.apache.hudi.keygen.factory.HoodieSparkKeyGeneratorFactory;
-import org.apache.hudi.storage.StoragePath;
import org.apache.hudi.utilities.UtilHelpers;
import org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer;
@@ -115,7 +114,7 @@ public class HoodieTestSuiteJob {
SparkSession.builder().config(jsc.getConf()).enableHiveSupport().getOrCreate();
this.fs = HadoopFSUtils.getFs(cfg.inputBasePath,
jsc.hadoopConfiguration());
this.props =
-UtilHelpers.readConfig(fs.getConf(), new
StoragePath(cfg.propsFilePath), cfg.configs).getProps();
+UtilHelpers.readConfig(fs.getConf(), new Path(cfg.propsFilePath),
cfg.configs).getProps();
log.info("Creating workload generator with configs : {}",
props.toString());
this.hiveConf = getDefaultHiveConf(jsc.hadoopConfiguration());
this.keyGenerator =
diff --git
a/hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/SparkDataSourceContinuousIngestTool.java
b/hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/SparkDataSourceContinuousIngestTool.java
index cbb2a27e54f..81bc4435623 100644
---
Re: [PR] [HUDI-7790] Revert changes in DFSPathSelector and UtilHelpers.readConfig [hudi]
yihua merged PR #11293: URL: https://github.com/apache/hudi/pull/11293 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7790] Revert changes in DFSPathSelector and UtilHelpers.readConfig [hudi]
yihua commented on PR #11293: URL: https://github.com/apache/hudi/pull/11293#issuecomment-2130745935 Azure CI is green. https://github.com/apache/hudi/assets/2497195/84505240-ebee-4406-847d-8623e08791b7;> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] [SUPPORT] Failed to parse HoodieCommitMetadata [hudi]
MrAladdin opened a new issue, #11299: URL: https://github.com/apache/hudi/issues/11299 **Describe the problem you faced** 1、Why and how to solve it **Environment Description** * Hudi version :0.14.1 * Spark version :3.4 * Hive version :3.1.2 * Hadoop version :3.1 * Storage (HDFS/S3/GCS..) :hdfs * Running on Docker? (yes/no) :no **Stacktrace** Caused by: org.apache.hudi.exception.HoodieIOException: Failed to parse HoodieCommitMetadata for [20240524230354074__deltacommit__COMPLETED__20240524231805574] at org.apache.hudi.common.util.CommitUtils.lambda$getValidCheckpointForCurrentWriter$3(CommitUtils.java:181) at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) at java.util.stream.SortedOps$SizedRefSortingSink.end(SortedOps.java:361) at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:500) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:486) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) at java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:152) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:531) at org.apache.hudi.common.util.CommitUtils.getValidCheckpointForCurrentWriter(CommitUtils.java:183) at org.apache.hudi.HoodieStreamingSink.canSkipBatch(HoodieStreamingSink.scala:325) at org.apache.hudi.HoodieStreamingSink.addBatch(HoodieStreamingSink.scala:108) at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runBatch$17(MicroBatchExecution.scala:729) at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:118) at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:195) at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:103) at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:827) at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:65) at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runBatch$16(MicroBatchExecution.scala:726) at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken(ProgressReporter.scala:411) at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken$(ProgressReporter.scala:409) at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:67) at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runBatch(MicroBatchExecution.scala:726) at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runActivatedStream$2(MicroBatchExecution.scala:284) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken(ProgressReporter.scala:411) at org.apache.spark.sql.execution.streaming.ProgressReporter.reportTimeTaken$(ProgressReporter.scala:409) at org.apache.spark.sql.execution.streaming.StreamExecution.reportTimeTaken(StreamExecution.scala:67) at org.apache.spark.sql.execution.streaming.MicroBatchExecution.$anonfun$runActivatedStream$1(MicroBatchExecution.scala:247) at org.apache.spark.sql.execution.streaming.ProcessingTimeExecutor.execute(TriggerExecutor.scala:67) at org.apache.spark.sql.execution.streaming.MicroBatchExecution.runActivatedStream(MicroBatchExecution.scala:237) at org.apache.spark.sql.execution.streaming.StreamExecution.$anonfun$runStream$1(StreamExecution.scala:306) at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:827) at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:284) ... 1 more Caused by: java.io.IOException: unable to read commit metadata at org.apache.hudi.common.model.HoodieCommitMetadata.fromBytes(HoodieCommitMetadata.java:514) at org.apache.hudi.common.util.CommitUtils.lambda$getValidCheckpointForCurrentWriter$3(CommitUtils.java:171) ... 35 more Caused by: com.fasterxml.jackson.databind.JsonMappingException: Unexpected end-of-input: was expecting closing quote for a string value at [Source: (StringReader); line: 3515, column: 41] (through reference chain: org.apache.hudi.common.model.HoodieCommitMetadata["partitionToWriteStats"]->java.util.LinkedHashMap["partition_index_date=202311"]->java.util.ArrayList[100]->org.apache.hudi.common.model.HoodieWriteStat["partitionPath"]) at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:402) at
Re: [I] [SUPPORT] Flink bucket index partitioner may cause data skew [hudi]
danny0405 commented on issue #11288: URL: https://github.com/apache/hudi/issues/11288#issuecomment-2130654910 The `parallelism/bucket_num` could be 0 when parallelism < bucket_num. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7794] Bump org.apache.hive:hive-service from 2.3.1 to 2.3.3 [hudi]
hudi-bot commented on PR #11298: URL: https://github.com/apache/hudi/pull/11298#issuecomment-2130654813 ## CI report: * f9ec7be4925a0b897c5b0dfab166d6905a495cbb UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7795] Fix loading of input splits from look up table reader [hudi]
hudi-bot commented on PR #11297: URL: https://github.com/apache/hudi/pull/11297#issuecomment-2130654798 ## CI report: * b890368f0e4246fd7b9982181b8a3762a25e1249 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7792] Bump h2 from 1.4.200 to 2.2.220 [hudi]
hudi-bot commented on PR #11296: URL: https://github.com/apache/hudi/pull/11296#issuecomment-2130654773 ## CI report: * d15db3e3e3f09197748317ed005b97dcf320d37d UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [HUDI-7794] Bump org.apache.hive:hive-service from 2.3.1 to 2.3.3 [hudi]
yihua opened a new pull request, #11298: URL: https://github.com/apache/hudi/pull/11298 ### Change Logs Bumps org.apache.hive:hive-service from 2.3.1 to 2.3.3. [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/hudi/network/alerts). ### Impact Fixes critical severity alert ### Risk level none ### Documentation Update none ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Change Logs and Impact were stated clearly - [ ] Adequate tests were added if applicable - [ ] CI passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7794] Bump org.apache.hive:hive-service from 2.3.1 to 2.3.3 [hudi]
hudi-bot commented on PR #11295: URL: https://github.com/apache/hudi/pull/11295#issuecomment-2130650728 ## CI report: * da65366d6498adea6a48d1b1c37701ed5912cd48 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-7794) Bump org.apache.hive:hive-service from 2.3.1 to 2.3.3
[ https://issues.apache.org/jira/browse/HUDI-7794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-7794: - Labels: pull-request-available (was: ) > Bump org.apache.hive:hive-service from 2.3.1 to 2.3.3 > - > > Key: HUDI-7794 > URL: https://issues.apache.org/jira/browse/HUDI-7794 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Assignee: Ethan Guo >Priority: Major > Labels: pull-request-available > Fix For: 0.15.0, 1.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HUDI-7795) Fix loading of input splits from look up table reader
[ https://issues.apache.org/jira/browse/HUDI-7795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-7795: - Labels: pull-request-available (was: ) > Fix loading of input splits from look up table reader > - > > Key: HUDI-7795 > URL: https://issues.apache.org/jira/browse/HUDI-7795 > Project: Apache Hudi > Issue Type: Improvement > Components: writer-core >Reporter: Danny Chen >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [HUDI-7795] Fix loading of input splits from look up table reader [hudi]
danny0405 opened a new pull request, #11297: URL: https://github.com/apache/hudi/pull/11297 ### Change Logs Should load all the input splits of the table. ### Impact none ### Risk level (write none, low medium or high below) none ### Documentation Update none ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Change Logs and Impact were stated clearly - [ ] Adequate tests were added if applicable - [ ] CI passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HUDI-7795) Fix loading of input splits from look up table reader
Danny Chen created HUDI-7795: Summary: Fix loading of input splits from look up table reader Key: HUDI-7795 URL: https://issues.apache.org/jira/browse/HUDI-7795 Project: Apache Hudi Issue Type: Improvement Components: writer-core Reporter: Danny Chen -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [HUDI-7792] Bump h2 from 1.4.200 to 2.2.220 [hudi]
yihua opened a new pull request, #11296: URL: https://github.com/apache/hudi/pull/11296 ### Change Logs Bumps [h2](https://github.com/h2database/h2database) from 1.4.200 to 2.2.220. Release notes Sourced from https://github.com/h2database/h2database/releases;>h2's releases. Version 2.2.220 Changes since 2.1.214 release: ... (truncated) Commits https://github.com/h2database/h2database/commit/66185fbfde24fbe5b2adf06fdaae4f5ffb178813;>66185fb changelog adjustment https://github.com/h2database/h2database/commit/f3c8222f3d77a8e71db2e6a72ea183c1931bfa6d;>f3c8222 version and release date https://github.com/h2database/h2database/commit/087522bbc84c1b26417dd1ea348d8fbf839d5a32;>087522b javadoc https://github.com/h2database/h2database/commit/1ae052a1c0c87c687ac1d312e2bd69a4def652aa;>1ae052a spell-check https://github.com/h2database/h2database/commit/4ac534343ff7bde11bda681da39075cfa34ca440;>4ac5343 Merge pull request https://redirect.github.com/h2database/h2database/issues/3834;>#3834 from katzyn/version https://github.com/h2database/h2database/commit/72f6e983789b0ff413ba110d218c18ded50cb7b8;>72f6e98 Update TestMVStore https://github.com/h2database/h2database/commit/81bac50fed2f8418d83455e8d4c7fbf855bede89;>81bac50 Update changelog https://github.com/h2database/h2database/commit/f047d77085d62cd1867edcf394944de46550d58a;>f047d77 Add 2.0.* and 2.1.* versions to Upgrade utility https://github.com/h2database/h2database/commit/1341fea79e0d82d5967ecd19fba206a740870a69;>1341fea Increase database format version https://github.com/h2database/h2database/commit/581ed18ff9d6b3761d851620ed88a3994a351a0d;>581ed18 Merge pull request https://redirect.github.com/h2database/h2database/issues/3833;>#3833 from katzyn/password Additional commits viewable in https://github.com/h2database/h2database/compare/version-1.4.200...version-2.2.220;>compare view [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) You can trigger a rebase of this PR by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/hudi/network/alerts). > **Note** > Automatic rebases have been disabled on this pull request as it has been open for over 30 days. ### Impact Fixes critical severity alert ### Risk level none ### Documentation Update none ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Change Logs and Impact were stated clearly - [ ] Adequate tests were added if applicable - [ ] CI passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(hudi) branch dependabot/maven/hudi-platform-service/hudi-metaserver/com.h2database-h2-2.2.220 deleted (was 1560e9aa30c)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/hudi-platform-service/hudi-metaserver/com.h2database-h2-2.2.220 in repository https://gitbox.apache.org/repos/asf/hudi.git was 1560e9aa30c Bump h2 in /hudi-platform-service/hudi-metaserver The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(hudi) branch master updated: Bump h2 in /hudi-platform-service/hudi-metaserver (#9147)
This is an automated email from the ASF dual-hosted git repository.
sivabalan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/master by this push:
new a0d60d98976 Bump h2 in /hudi-platform-service/hudi-metaserver (#9147)
a0d60d98976 is described below
commit a0d60d98976d53958e1e04352d2b3b714c395e9c
Author: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
AuthorDate: Fri May 24 18:23:14 2024 -0700
Bump h2 in /hudi-platform-service/hudi-metaserver (#9147)
Bumps [h2](https://github.com/h2database/h2database) from 1.4.200 to
2.2.220.
- [Release notes](https://github.com/h2database/h2database/releases)
-
[Commits](https://github.com/h2database/h2database/compare/version-1.4.200...version-2.2.220)
---
updated-dependencies:
- dependency-name: com.h2database:h2
dependency-type: direct:production
...
Signed-off-by: dependabot[bot]
Co-authored-by: dependabot[bot]
<49699333+dependabot[bot]@users.noreply.github.com>
---
hudi-platform-service/hudi-metaserver/pom.xml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/hudi-platform-service/hudi-metaserver/pom.xml
b/hudi-platform-service/hudi-metaserver/pom.xml
index 38b25533fe5..b42a229fbd0 100644
--- a/hudi-platform-service/hudi-metaserver/pom.xml
+++ b/hudi-platform-service/hudi-metaserver/pom.xml
@@ -32,7 +32,7 @@
${project.parent.basedir}
-1.4.200
+2.2.220
/usr/local
docker
Re: [PR] [HUDI-7792] Bump h2 from 1.4.200 to 2.2.220 in /hudi-platform-service/hudi-metaserver [hudi]
nsivabalan merged PR #9147: URL: https://github.com/apache/hudi/pull/9147 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(hudi) branch dependabot/maven/packaging/hudi-metaserver-server-bundle/com.h2database-h2-2.2.220 deleted (was cb331f02ad6)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/packaging/hudi-metaserver-server-bundle/com.h2database-h2-2.2.220 in repository https://gitbox.apache.org/repos/asf/hudi.git was cb331f02ad6 Bump h2 in /packaging/hudi-metaserver-server-bundle The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(hudi) branch master updated (660c6c7a5b0 -> 70a04c6c782)
This is an automated email from the ASF dual-hosted git repository. sivabalan pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from 660c6c7a5b0 [MINOR] Fix the look up warning msg (#11286) add 70a04c6c782 Bump h2 in /packaging/hudi-metaserver-server-bundle (#9148) No new revisions were added by this update. Summary of changes: packaging/hudi-metaserver-server-bundle/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Re: [PR] [HUDI-7791] Bump h2 from 1.4.200 to 2.2.220 in /packaging/hudi-metaserver-server-bundle [hudi]
nsivabalan merged PR #9148: URL: https://github.com/apache/hudi/pull/9148 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7776] Simplify HoodieStorage instance fetching [hudi]
yihua commented on PR #11292: URL: https://github.com/apache/hudi/pull/11292#issuecomment-2130638254 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-7792) Bump h2 from 1.4.200 to 2.2.220 in /hudi-platform-service/hudi-metaserver
[ https://issues.apache.org/jira/browse/HUDI-7792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-7792: - Labels: pull-request-available (was: ) > Bump h2 from 1.4.200 to 2.2.220 in /hudi-platform-service/hudi-metaserver > - > > Key: HUDI-7792 > URL: https://issues.apache.org/jira/browse/HUDI-7792 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Assignee: Ethan Guo >Priority: Major > Labels: pull-request-available > Fix For: 0.15.0, 1.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HUDI-7791) Bump h2 from 1.4.200 to 2.2.220 in /packaging/hudi-metaserver-server-bundle
[ https://issues.apache.org/jira/browse/HUDI-7791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-7791: - Labels: pull-request-available (was: ) > Bump h2 from 1.4.200 to 2.2.220 in /packaging/hudi-metaserver-server-bundle > --- > > Key: HUDI-7791 > URL: https://issues.apache.org/jira/browse/HUDI-7791 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Assignee: Ethan Guo >Priority: Major > Labels: pull-request-available > Fix For: 0.15.0, 1.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [HUDI-7792] Bump h2 from 1.4.200 to 2.2.220 in /hudi-platform-service/hudi-metaserver [hudi]
yihua commented on PR #9147: URL: https://github.com/apache/hudi/pull/9147#issuecomment-2130636389 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7791] Bump h2 from 1.4.200 to 2.2.220 in /packaging/hudi-metaserver-server-bundle [hudi]
yihua commented on PR #9148: URL: https://github.com/apache/hudi/pull/9148#issuecomment-2130636375 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HUDI-7794) Bump org.apache.hive:hive-service from 2.3.1 to 2.3.3
Ethan Guo created HUDI-7794: --- Summary: Bump org.apache.hive:hive-service from 2.3.1 to 2.3.3 Key: HUDI-7794 URL: https://issues.apache.org/jira/browse/HUDI-7794 Project: Apache Hudi Issue Type: Improvement Reporter: Ethan Guo -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HUDI-7794) Bump org.apache.hive:hive-service from 2.3.1 to 2.3.3
[ https://issues.apache.org/jira/browse/HUDI-7794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo reassigned HUDI-7794: --- Assignee: Ethan Guo > Bump org.apache.hive:hive-service from 2.3.1 to 2.3.3 > - > > Key: HUDI-7794 > URL: https://issues.apache.org/jira/browse/HUDI-7794 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Assignee: Ethan Guo >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HUDI-7794) Bump org.apache.hive:hive-service from 2.3.1 to 2.3.3
[ https://issues.apache.org/jira/browse/HUDI-7794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7794: Fix Version/s: 0.15.0 1.0.0 > Bump org.apache.hive:hive-service from 2.3.1 to 2.3.3 > - > > Key: HUDI-7794 > URL: https://issues.apache.org/jira/browse/HUDI-7794 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Assignee: Ethan Guo >Priority: Major > Fix For: 0.15.0, 1.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HUDI-7793) Upgrade Hive version from 2.3.1 to 2.3.4
Ethan Guo created HUDI-7793: --- Summary: Upgrade Hive version from 2.3.1 to 2.3.4 Key: HUDI-7793 URL: https://issues.apache.org/jira/browse/HUDI-7793 Project: Apache Hudi Issue Type: Improvement Reporter: Ethan Guo -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] Bump hive.version from 2.3.1 to 3.1.2 [hudi]
yihua commented on PR #983: URL: https://github.com/apache/hudi/pull/983#issuecomment-2130630675 @dependabot reopen -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Bump hive.version from 2.3.1 to 3.1.2 [hudi]
dependabot[bot] closed pull request #983: Bump hive.version from 2.3.1 to 3.1.2 URL: https://github.com/apache/hudi/pull/983 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(hudi) branch dependabot/maven/hive.version-3.1.2 deleted (was ac3ae4a66fa)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/hive.version-3.1.2 in repository https://gitbox.apache.org/repos/asf/hudi.git was ac3ae4a66fa Merge branch 'master' into dependabot/maven/hive.version-3.1.2 The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
Re: [PR] Bump hive.version from 2.3.1 to 3.1.2 [hudi]
dependabot[bot] commented on PR #983: URL: https://github.com/apache/hudi/pull/983#issuecomment-2130630850 Looks like these dependencies are updatable in another way, so this is no longer needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(hudi) branch dependabot/maven/hive.version-3.1.2 created (now 14e3d559bd0)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/hive.version-3.1.2 in repository https://gitbox.apache.org/repos/asf/hudi.git at 14e3d559bd0 Bump hive.version from 2.3.1 to 3.1.2 No new revisions were added by this update.
(hudi) branch dependabot/maven/hive.version-3.1.2 deleted (was 14e3d559bd0)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/hive.version-3.1.2 in repository https://gitbox.apache.org/repos/asf/hudi.git was 14e3d559bd0 Bump hive.version from 2.3.1 to 3.1.2 The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
Re: [PR] Bump hive.version from 2.3.1 to 3.1.2 [hudi]
dependabot[bot] closed pull request #983: Bump hive.version from 2.3.1 to 3.1.2 URL: https://github.com/apache/hudi/pull/983 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Bump hive.version from 2.3.1 to 3.1.2 [hudi]
dependabot[bot] commented on PR #983: URL: https://github.com/apache/hudi/pull/983#issuecomment-2130630126 Looks like these dependencies are updatable in another way, so this is no longer needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(hudi) branch dependabot/maven/hive.version-3.1.2 created (now 14e3d559bd0)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/hive.version-3.1.2 in repository https://gitbox.apache.org/repos/asf/hudi.git at 14e3d559bd0 Bump hive.version from 2.3.1 to 3.1.2 No new revisions were added by this update.
Re: [PR] Bump hive.version from 2.3.1 to 3.1.2 [hudi]
yihua commented on PR #983: URL: https://github.com/apache/hudi/pull/983#issuecomment-2130629909 @dependabot recreate -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Bump hive.version from 2.3.1 to 3.1.2 [hudi]
yihua commented on PR #983: URL: https://github.com/apache/hudi/pull/983#issuecomment-2130629887 @dependabot reopen -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(hudi) branch master updated: [MINOR] Fix the look up warning msg (#11286)
This is an automated email from the ASF dual-hosted git repository.
danny0405 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/master by this push:
new 660c6c7a5b0 [MINOR] Fix the look up warning msg (#11286)
660c6c7a5b0 is described below
commit 660c6c7a5b060af20b152b49fcc07bd4482e127d
Author: leixin <1403342...@qq.com>
AuthorDate: Sat May 25 08:56:44 2024 +0800
[MINOR] Fix the look up warning msg (#11286)
Co-authored-by: leixin1
---
.../src/main/java/org/apache/hudi/table/HoodieTableSource.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableSource.java
b/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableSource.java
index 9726c1d564a..9a431594eb8 100644
---
a/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableSource.java
+++
b/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableSource.java
@@ -616,7 +616,7 @@ public class HoodieTableSource implements
for (int[] key : keys) {
if (key.length > 1) {
throw new UnsupportedOperationException(
-"Hive lookup can not support nested key now.");
+"Hoodie lookup can not support nested key now.");
}
keyIndices[i] = key[0];
i++;
Re: [PR] [minor] rename to hoodie [hudi]
danny0405 merged PR #11286: URL: https://github.com/apache/hudi/pull/11286 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Bump hive.version from 2.3.1 to 3.1.2 [hudi]
dependabot[bot] commented on PR #983: URL: https://github.com/apache/hudi/pull/983#issuecomment-2130628379 Looks like this PR is closed. If you re-open it I'll rebase it as long as no-one else has edited it (you can use `@dependabot reopen` if the branch has been deleted). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Bump hive.version from 2.3.1 to 3.1.2 [hudi]
yihua commented on PR #983: URL: https://github.com/apache/hudi/pull/983#issuecomment-2130628333 @dependabot recreate -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(hudi) branch dependabot/maven/org.apache.hive-hive-service-2.3.3 created (now da65366d649)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/org.apache.hive-hive-service-2.3.3 in repository https://gitbox.apache.org/repos/asf/hudi.git at da65366d649 Bump org.apache.hive:hive-service from 2.3.1 to 2.3.3 No new revisions were added by this update.
(hudi) branch dependabot/maven/hive.version-3.1.2 deleted (was 14e3d559bd0)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/hive.version-3.1.2 in repository https://gitbox.apache.org/repos/asf/hudi.git was 14e3d559bd0 Bump hive.version from 2.3.1 to 3.1.2 The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(hudi) branch dependabot/maven/hive.version-3.1.2 created (now 14e3d559bd0)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/hive.version-3.1.2 in repository https://gitbox.apache.org/repos/asf/hudi.git at 14e3d559bd0 Bump hive.version from 2.3.1 to 3.1.2 No new revisions were added by this update.
(hudi) branch master updated: [HUDI-7787] Ignore loading data for look up function when the commit has not changed (#11287)
This is an automated email from the ASF dual-hosted git repository.
danny0405 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/master by this push:
new 376571af371 [HUDI-7787] Ignore loading data for look up function when
the commit has not changed (#11287)
376571af371 is described below
commit 376571af3719834739599f698673f185ac409830
Author: hehuiyuan <471627...@qq.com>
AuthorDate: Sat May 25 08:54:26 2024 +0800
[HUDI-7787] Ignore loading data for look up function when the commit has
not changed (#11287)
---
.../org/apache/hudi/table/HoodieTableSource.java | 3 ++-
.../hudi/table/lookup/HoodieLookupFunction.java| 31 +-
2 files changed, 32 insertions(+), 2 deletions(-)
diff --git
a/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableSource.java
b/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableSource.java
index 0e3749a493f..9726c1d564a 100644
---
a/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableSource.java
+++
b/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/HoodieTableSource.java
@@ -310,7 +310,8 @@ public class HoodieTableSource implements
new HoodieLookupTableReader(this::getBatchInputFormat, conf),
tableRowType,
getLookupKeys(context.getKeys()),
-duration
+duration,
+conf
));
}
diff --git
a/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/lookup/HoodieLookupFunction.java
b/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/lookup/HoodieLookupFunction.java
index 3fc65ce12b4..fe32a439f72 100644
---
a/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/lookup/HoodieLookupFunction.java
+++
b/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/lookup/HoodieLookupFunction.java
@@ -18,9 +18,17 @@
package org.apache.hudi.table.lookup;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.configuration.HadoopConfigurations;
+import org.apache.hudi.util.StreamerUtil;
+
import org.apache.flink.annotation.VisibleForTesting;
import org.apache.flink.api.common.typeinfo.TypeInformation;
import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.configuration.Configuration;
import org.apache.flink.table.data.GenericRowData;
import org.apache.flink.table.data.RowData;
import org.apache.flink.table.functions.FunctionContext;
@@ -64,11 +72,16 @@ public class HoodieLookupFunction extends
TableFunction {
// timestamp when cache expires
private transient long nextLoadTime;
+ private transient HoodieTableMetaClient metaClient;
+ private transient HoodieInstant currentCommit;
+ private final Configuration conf;
+
public HoodieLookupFunction(
HoodieLookupTableReader partitionReader,
RowType rowType,
int[] lookupKeys,
- Duration reloadInterval) {
+ Duration reloadInterval,
+ Configuration conf) {
this.partitionReader = partitionReader;
this.rowType = rowType;
this.lookupFieldGetters = new RowData.FieldGetter[lookupKeys.length];
@@ -78,6 +91,7 @@ public class HoodieLookupFunction extends
TableFunction {
}
this.reloadInterval = reloadInterval;
this.serializer = InternalSerializers.create(rowType);
+this.conf = conf;
}
@Override
@@ -85,6 +99,8 @@ public class HoodieLookupFunction extends
TableFunction {
super.open(context);
cache = new HashMap<>();
nextLoadTime = -1L;
+org.apache.hadoop.conf.Configuration hadoopConf =
HadoopConfigurations.getHadoopConf(conf);
+metaClient = StreamerUtil.metaClientForReader(conf, hadoopConf);
}
@Override
@@ -114,6 +130,19 @@ public class HoodieLookupFunction extends
TableFunction {
} else {
LOG.info("Populating lookup join cache");
}
+
+HoodieActiveTimeline latestCommit = metaClient.reloadActiveTimeline();
+Option latestCommitInstant =
latestCommit.getCommitsTimeline().lastInstant();
+if (latestCommit.empty()) {
+ LOG.info("No commit instant found currently.");
+ return;
+}
+// Determine whether to reload data by comparing instant
+if (currentCommit != null &&
latestCommitInstant.get().equals(currentCommit)) {
+ LOG.info("Ignore loading data because the commit instant " +
currentCommit + " has not changed.");
+ return;
+}
+
int numRetry = 0;
while (true) {
cache.clear();
[PR] Bump org.apache.hive:hive-service from 2.3.1 to 2.3.3 [hudi]
dependabot[bot] opened a new pull request, #11295: URL: https://github.com/apache/hudi/pull/11295 Bumps org.apache.hive:hive-service from 2.3.1 to 2.3.3. [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually - `@dependabot show ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/hudi/network/alerts). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Bump hive.version from 2.3.1 to 3.1.2 [hudi]
dependabot[bot] closed pull request #983: Bump hive.version from 2.3.1 to 3.1.2 URL: https://github.com/apache/hudi/pull/983 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Bump hive.version from 2.3.1 to 3.1.2 [hudi]
dependabot[bot] commented on PR #983: URL: https://github.com/apache/hudi/pull/983#issuecomment-2130628238 Looks like these dependencies are updatable in another way, so this is no longer needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-7787) Reload the data for lookup table when found the newer commit instance
[ https://issues.apache.org/jira/browse/HUDI-7787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated HUDI-7787: - Fix Version/s: 1.0.0 > Reload the data for lookup table when found the newer commit instance > - > > Key: HUDI-7787 > URL: https://issues.apache.org/jira/browse/HUDI-7787 > Project: Apache Hudi > Issue Type: Improvement >Reporter: hehuiyuan >Priority: Major > Labels: pull-request-available > Fix For: 1.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (HUDI-7787) Reload the data for lookup table when found the newer commit instance
[ https://issues.apache.org/jira/browse/HUDI-7787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen closed HUDI-7787. Resolution: Fixed Fixed via master branch: 376571af3719834739599f698673f185ac409830 > Reload the data for lookup table when found the newer commit instance > - > > Key: HUDI-7787 > URL: https://issues.apache.org/jira/browse/HUDI-7787 > Project: Apache Hudi > Issue Type: Improvement >Reporter: hehuiyuan >Priority: Major > Labels: pull-request-available > Fix For: 1.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] Bump hive.version from 2.3.1 to 3.1.2 [hudi]
yihua commented on PR #983: URL: https://github.com/apache/hudi/pull/983#issuecomment-2130628165 https://github.com/dependabot recreate -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Bump hive.version from 2.3.1 to 3.1.2 [hudi]
dependabot[bot] opened a new pull request, #983: URL: https://github.com/apache/hudi/pull/983 Bumps `hive.version` from 2.3.1 to 3.1.2. Updates `hive-service` from 2.3.1 to 3.1.2 Updates `hive-shims` from 2.3.1 to 3.1.2 Updates `hive-jdbc` from 2.3.1 to 3.1.2 Updates `hive-serde` from 2.3.1 to 3.1.2 Updates `hive-metastore` from 2.3.1 to 3.1.2 Updates `hive-common` from 2.3.1 to 3.1.2 Updates `hive-exec` from 2.3.1 to 3.1.2 Updates `hive-service-rpc` from 2.3.1 to 3.1.2 Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- Dependabot commands and options You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot merge` will merge this PR after your CI passes on it - `@dependabot squash and merge` will squash and merge this PR after your CI passes on it - `@dependabot cancel merge` will cancel a previously requested merge and block automerging - `@dependabot reopen` will reopen this PR if it is closed - `@dependabot ignore this [patch|minor|major] version` will close this PR and stop Dependabot creating any more for this minor/major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) - `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language - `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language - `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language - `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/apache/incubator-hudi/network/alerts). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Bump hive.version from 2.3.1 to 3.1.2 [hudi]
yihua commented on PR #983: URL: https://github.com/apache/hudi/pull/983#issuecomment-2130628086 @dependabot reopen -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7787] Ignore loading data when the commit has not changed [hudi]
danny0405 merged PR #11287: URL: https://github.com/apache/hudi/pull/11287 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7787] Ignore loading data when the commit has not changed [hudi]
danny0405 commented on PR #11287: URL: https://github.com/apache/hudi/pull/11287#issuecomment-2130627820 Azure CI passed: https://dev.azure.com/apache-hudi-ci-org/apache-hudi-ci/_build/results?buildId=24034=results -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Bump hive.version from 2.3.1 to 3.1.2 [hudi]
dependabot[bot] commented on PR #983: URL: https://github.com/apache/hudi/pull/983#issuecomment-2130627953 Looks like this PR is closed. If you re-open it I'll rebase it as long as no-one else has edited it (you can use `@dependabot reopen` if the branch has been deleted). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Bump hive.version from 2.3.1 to 3.1.2 [hudi]
yihua commented on PR #983: URL: https://github.com/apache/hudi/pull/983#issuecomment-2130627938 @dependabot recreate -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-7792) Bump h2 from 1.4.200 to 2.2.220 in /hudi-platform-service/hudi-metaserver
[ https://issues.apache.org/jira/browse/HUDI-7792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7792: Fix Version/s: 0.15.0 1.0.0 > Bump h2 from 1.4.200 to 2.2.220 in /hudi-platform-service/hudi-metaserver > - > > Key: HUDI-7792 > URL: https://issues.apache.org/jira/browse/HUDI-7792 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Assignee: Ethan Guo >Priority: Major > Fix For: 0.15.0, 1.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HUDI-7792) Bump h2 from 1.4.200 to 2.2.220 in /hudi-platform-service/hudi-metaserver
[ https://issues.apache.org/jira/browse/HUDI-7792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo reassigned HUDI-7792: --- Assignee: Ethan Guo > Bump h2 from 1.4.200 to 2.2.220 in /hudi-platform-service/hudi-metaserver > - > > Key: HUDI-7792 > URL: https://issues.apache.org/jira/browse/HUDI-7792 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Assignee: Ethan Guo >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HUDI-7791) Bump h2 from 1.4.200 to 2.2.220 in /packaging/hudi-metaserver-server-bundle
[ https://issues.apache.org/jira/browse/HUDI-7791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo reassigned HUDI-7791: --- Assignee: Ethan Guo > Bump h2 from 1.4.200 to 2.2.220 in /packaging/hudi-metaserver-server-bundle > --- > > Key: HUDI-7791 > URL: https://issues.apache.org/jira/browse/HUDI-7791 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Assignee: Ethan Guo >Priority: Major > Fix For: 0.15.0, 1.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HUDI-7791) Bump h2 from 1.4.200 to 2.2.220 in /packaging/hudi-metaserver-server-bundle
[ https://issues.apache.org/jira/browse/HUDI-7791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7791: Fix Version/s: 0.15.0 1.0.0 > Bump h2 from 1.4.200 to 2.2.220 in /packaging/hudi-metaserver-server-bundle > --- > > Key: HUDI-7791 > URL: https://issues.apache.org/jira/browse/HUDI-7791 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Priority: Major > Fix For: 0.15.0, 1.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HUDI-7792) Bump h2 from 1.4.200 to 2.2.220 in /hudi-platform-service/hudi-metaserver
Ethan Guo created HUDI-7792: --- Summary: Bump h2 from 1.4.200 to 2.2.220 in /hudi-platform-service/hudi-metaserver Key: HUDI-7792 URL: https://issues.apache.org/jira/browse/HUDI-7792 Project: Apache Hudi Issue Type: Improvement Reporter: Ethan Guo -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HUDI-7791) Bump h2 from 1.4.200 to 2.2.220 in /packaging/hudi-metaserver-server-bundle
Ethan Guo created HUDI-7791: --- Summary: Bump h2 from 1.4.200 to 2.2.220 in /packaging/hudi-metaserver-server-bundle Key: HUDI-7791 URL: https://issues.apache.org/jira/browse/HUDI-7791 Project: Apache Hudi Issue Type: Improvement Reporter: Ethan Guo -- This message was sent by Atlassian Jira (v8.20.10#820010)
(hudi) branch dependabot/maven/hudi-platform-service/hudi-metaserver/com.h2database-h2-2.2.220 updated (e96f60a4406 -> 1560e9aa30c)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/hudi-platform-service/hudi-metaserver/com.h2database-h2-2.2.220 in repository https://gitbox.apache.org/repos/asf/hudi.git discard e96f60a4406 Bump h2 in /hudi-platform-service/hudi-metaserver add ddaef8feddb [HUDI-5101] Adding spark-structured streaming test support via spark-submit job (#7074) add e2dfb465f13 [HUDI-7495] Bump mysql-connector-java from 8.0.22 to 8.0.28 in /hudi-platform-service/hudi-metaserver/hudi-metaserver-server (#7674) add e6664159bed [HUDI-7163] Fix not parsable text DateTimeParseException when compact (#10220) add 3698d49383b [HUDI-7496] Bump mybatis from 3.4.6 to 3.5.6 in /hudi-platform-service/hudi-metaserver/hudi-metaserver-server (#7673) add 819788f8651 [MINOR] Remove repetitive words in docs (#10844) add ee11b9c951c [HUDI-7489] Avoid collecting WriteStatus to driver in row writer code path (#10836) add 130498708bb add job context (#10848) add 8bc9a4bc875 [HUDI-7478] Fix max delta commits guard check w/ MDT (#10820) add f8f12ba9ef3 [MINOR] Fix and enable test TestHoodieDeltaStreamer.testJdbcSourceIncrementalFetchInContinuousMode (#10867) add aac6b2e5486 [HUDI-7382] Get partitions from active timeline instead of listing when building clustering plan (#10621) add ca2140e2003 [MINOR] rename KeyGenUtils#enableAutoGenerateRecordKeys (#10871) add 3c8488b831c [HUDI-7506] Compute offsetRanges based on eventsPerPartition allocated in each range (#10869) add e726306cf09 [HUDI-7466] Add parallel listing of existing partitions in Glue Catalog sync (#10460) add 2dcdd311245 [HUDI-7421] Build HoodieDeltaWriteStat using HoodieDeltaWriteStat#copy (#10870) add b7ccecf3205 [HUDI-7492] Fix the incorrect keygenerator specification for multi partition or multi primary key tables creation (#10840) add 7631e0dcb89 [MINOR] Add Hudi icon for idea (#10880) add 784af0e1786 [HUDI-7514] Update Manifest file after the parquet writer closed in LSMTimelineWriter (#10883) add 7c55ac35ba1 [HUDI-7516] Put jdbc-h2 creds into static variables for hudi-utilities tests (#10889) add 135db099afc [MINOR] Remove redundant fileId from HoodieAppendHandle (#10901) add 5a21a1dd260 [HUDI-7529] Resolve hotspots in stream read (#10911) add 47151f653d8 [HUDI-7487] Fixed test with in-memory index by proper heap clearing (#10910) add 6be7205a1e3 [MINOR] Refactored `@Before*` and `@After*` in `HoodieDeltaStreamerTestBase` (#10912) add a8e9db446c3 [HUDI-7530] Refactoring of handleUpdateInternal in CommitActionExecutors and HoodieTables (#10908) add f98a40bd369 [HUDI-7499] Support FirstValueAvroPayload for Hudi (#10857) add da9660bf38a checkstyle (#10919) add d749457f9d5 [HUDI-7513] Add jackson-module-scala to spark bundle (#10877) add d22bfba08fd [MINOR] Restore the setMaxParallelism setting for HoodieTableSource.produceDataStream (#10925) add 5e4a6c650e0 [HUDI-7531] Consider pending clustering when scheduling a new clustering plan (#10923) add 8a137631da8 [HUDI-7518] Fix HoodieMetadataPayload merging logic around repeated deletes (#10913) add 136d0755ad7 [HUDI-7500] fix gaps with deduce schema and null schema (#10858) add 28f67ff3561 [HUDI-7551] Avoid loading all partitions in CleanPlanner when MDT is enabled (#10928) add 4741ba06462 [HUDI-6317] Streaming read should skip compaction and clustering instants to avoid duplicates (#8884) add ae4f46874a9 [MINOR} When M3 metrics reporter type is used HoodieMetricsConfig should create default values for HoodieMetricsM3Config (#10936) add 06d3bb8cfbd [HUDI-6884] hudi-cli should generate correct HoodieTimeGeneratorConfig (#10941) add 26c00a3adef [HUDI-7187] Fix integ test props to honor new streamer properties (#10866) add 9b094e628d6 [HUDI-7510] Loosen the compaction scheduling and rollback check for MDT (#10874) add 44ab6f32bff [HUDI-6538] Refactor methods in TimelineDiffHelper class (#10938) add 9efced37f81 [HUDI-7557] Fix incremental cleaner when commit for savepoint removed (#10946) add bb51aca75d0 [MINOR] Upgrade mockito to 3.12.4 (#10953) add 8bb6bee6234 [HUDI-7564] Fix HiveSyncConfig inconsistency (#10951) add 59e32b7e686 [HUDI-7569] [RLI] Fix wrong result generated by query (#10955) add bf723f56cd0 [HUDI-7486] Classify schema exceptions when converting from avro to spark row representation (#10778) add 398c9a23c84 [HUDI-7564] Revert hive sync inconsistency and reason for it (#10959) add 8b61696f158 [HUDI-7556] Fixing MDT validator and adding tests (#10939) add 19c20e4dd93 [HUDI-7571] Add api to get exception details in HoodieMetadataTableValidator with ignoreFailed mode (#10960) add bac6ea7b26b [MINOR] Removed FSUtils.makeBaseFileName without fileExt param (#10963) add d41541cb9f8
Re: [PR] Bump h2 from 1.4.200 to 2.2.220 in /packaging/hudi-metaserver-server-bundle [hudi]
yihua commented on PR #9148: URL: https://github.com/apache/hudi/pull/9148#issuecomment-2130615681 @dependabot recreate -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(hudi) branch dependabot/maven/packaging/hudi-metaserver-server-bundle/com.h2database-h2-2.2.220 updated (b10b98e5d8f -> cb331f02ad6)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/packaging/hudi-metaserver-server-bundle/com.h2database-h2-2.2.220 in repository https://gitbox.apache.org/repos/asf/hudi.git discard b10b98e5d8f Bump h2 in /packaging/hudi-metaserver-server-bundle add ddaef8feddb [HUDI-5101] Adding spark-structured streaming test support via spark-submit job (#7074) add e2dfb465f13 [HUDI-7495] Bump mysql-connector-java from 8.0.22 to 8.0.28 in /hudi-platform-service/hudi-metaserver/hudi-metaserver-server (#7674) add e6664159bed [HUDI-7163] Fix not parsable text DateTimeParseException when compact (#10220) add 3698d49383b [HUDI-7496] Bump mybatis from 3.4.6 to 3.5.6 in /hudi-platform-service/hudi-metaserver/hudi-metaserver-server (#7673) add 819788f8651 [MINOR] Remove repetitive words in docs (#10844) add ee11b9c951c [HUDI-7489] Avoid collecting WriteStatus to driver in row writer code path (#10836) add 130498708bb add job context (#10848) add 8bc9a4bc875 [HUDI-7478] Fix max delta commits guard check w/ MDT (#10820) add f8f12ba9ef3 [MINOR] Fix and enable test TestHoodieDeltaStreamer.testJdbcSourceIncrementalFetchInContinuousMode (#10867) add aac6b2e5486 [HUDI-7382] Get partitions from active timeline instead of listing when building clustering plan (#10621) add ca2140e2003 [MINOR] rename KeyGenUtils#enableAutoGenerateRecordKeys (#10871) add 3c8488b831c [HUDI-7506] Compute offsetRanges based on eventsPerPartition allocated in each range (#10869) add e726306cf09 [HUDI-7466] Add parallel listing of existing partitions in Glue Catalog sync (#10460) add 2dcdd311245 [HUDI-7421] Build HoodieDeltaWriteStat using HoodieDeltaWriteStat#copy (#10870) add b7ccecf3205 [HUDI-7492] Fix the incorrect keygenerator specification for multi partition or multi primary key tables creation (#10840) add 7631e0dcb89 [MINOR] Add Hudi icon for idea (#10880) add 784af0e1786 [HUDI-7514] Update Manifest file after the parquet writer closed in LSMTimelineWriter (#10883) add 7c55ac35ba1 [HUDI-7516] Put jdbc-h2 creds into static variables for hudi-utilities tests (#10889) add 135db099afc [MINOR] Remove redundant fileId from HoodieAppendHandle (#10901) add 5a21a1dd260 [HUDI-7529] Resolve hotspots in stream read (#10911) add 47151f653d8 [HUDI-7487] Fixed test with in-memory index by proper heap clearing (#10910) add 6be7205a1e3 [MINOR] Refactored `@Before*` and `@After*` in `HoodieDeltaStreamerTestBase` (#10912) add a8e9db446c3 [HUDI-7530] Refactoring of handleUpdateInternal in CommitActionExecutors and HoodieTables (#10908) add f98a40bd369 [HUDI-7499] Support FirstValueAvroPayload for Hudi (#10857) add da9660bf38a checkstyle (#10919) add d749457f9d5 [HUDI-7513] Add jackson-module-scala to spark bundle (#10877) add d22bfba08fd [MINOR] Restore the setMaxParallelism setting for HoodieTableSource.produceDataStream (#10925) add 5e4a6c650e0 [HUDI-7531] Consider pending clustering when scheduling a new clustering plan (#10923) add 8a137631da8 [HUDI-7518] Fix HoodieMetadataPayload merging logic around repeated deletes (#10913) add 136d0755ad7 [HUDI-7500] fix gaps with deduce schema and null schema (#10858) add 28f67ff3561 [HUDI-7551] Avoid loading all partitions in CleanPlanner when MDT is enabled (#10928) add 4741ba06462 [HUDI-6317] Streaming read should skip compaction and clustering instants to avoid duplicates (#8884) add ae4f46874a9 [MINOR} When M3 metrics reporter type is used HoodieMetricsConfig should create default values for HoodieMetricsM3Config (#10936) add 06d3bb8cfbd [HUDI-6884] hudi-cli should generate correct HoodieTimeGeneratorConfig (#10941) add 26c00a3adef [HUDI-7187] Fix integ test props to honor new streamer properties (#10866) add 9b094e628d6 [HUDI-7510] Loosen the compaction scheduling and rollback check for MDT (#10874) add 44ab6f32bff [HUDI-6538] Refactor methods in TimelineDiffHelper class (#10938) add 9efced37f81 [HUDI-7557] Fix incremental cleaner when commit for savepoint removed (#10946) add bb51aca75d0 [MINOR] Upgrade mockito to 3.12.4 (#10953) add 8bb6bee6234 [HUDI-7564] Fix HiveSyncConfig inconsistency (#10951) add 59e32b7e686 [HUDI-7569] [RLI] Fix wrong result generated by query (#10955) add bf723f56cd0 [HUDI-7486] Classify schema exceptions when converting from avro to spark row representation (#10778) add 398c9a23c84 [HUDI-7564] Revert hive sync inconsistency and reason for it (#10959) add 8b61696f158 [HUDI-7556] Fixing MDT validator and adding tests (#10939) add 19c20e4dd93 [HUDI-7571] Add api to get exception details in HoodieMetadataTableValidator with ignoreFailed mode (#10960) add bac6ea7b26b [MINOR] Removed FSUtils.makeBaseFileName without fileExt param (#10963) add d41541cb9f8
Re: [PR] Bump h2 from 1.4.200 to 2.2.220 in /hudi-platform-service/hudi-metaserver [hudi]
yihua commented on PR #9147: URL: https://github.com/apache/hudi/pull/9147#issuecomment-2130615496 @dependabot recreate -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7790] Revert changes in DFSPathSelector and UtilHelpers.readConfig [hudi]
hudi-bot commented on PR #11294: URL: https://github.com/apache/hudi/pull/11294#issuecomment-2130605099 ## CI report: * 03e9ba49998b64fca42b47fcd33e3c103f120ac1 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7790] Revert changes in DFSPathSelector and UtilHelpers.readConfig [hudi]
hudi-bot commented on PR #11293: URL: https://github.com/apache/hudi/pull/11293#issuecomment-2130604858 ## CI report: * 82c8061f78034840ed5e121c88201f47d2b562e4 UNKNOWN * 4e2ad1de59f01f16cb69ed49cce26c715e6aaf7a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7790] Revert changes in DFSPathSelector and UtilHelpers.readConfig [hudi]
hudi-bot commented on PR #11293: URL: https://github.com/apache/hudi/pull/11293#issuecomment-2130589172 ## CI report: * 82c8061f78034840ed5e121c88201f47d2b562e4 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [HUDI-7790] Revert changes in DFSPathSelector and UtilHelpers.readConfig [hudi]
yihua opened a new pull request, #11294: URL: https://github.com/apache/hudi/pull/11294 ### Change Logs PR targeting master: https://github.com/apache/hudi/pull/11293 This PR targets `branch-0.x` with the same changes for 0.x releases. This PR reverts the changes in `DFSPathSelector`, subclasses, and `UtilHelpers.readConfig` API so they still use Hadoop classes, given they are used in Spark and utilities classes, to avoid behavior change and breaking API change. ### Impact As above, makes upgrade easier. ### Risk level none ### Documentation Update none ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Change Logs and Impact were stated clearly - [ ] Adequate tests were added if applicable - [ ] CI passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-7790) Revert changes in DFSPathSelector and UtilHelpers.readConfig
[ https://issues.apache.org/jira/browse/HUDI-7790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-7790: - Labels: pull-request-available (was: ) > Revert changes in DFSPathSelector and UtilHelpers.readConfig > > > Key: HUDI-7790 > URL: https://issues.apache.org/jira/browse/HUDI-7790 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Assignee: Ethan Guo >Priority: Major > Labels: pull-request-available > Fix For: 0.15.0, 1.0.0 > > > This is to avoid behavior changes in DFSPathSelector and keep the > UtilHelpers.readConfig API the same as before. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [HUDI-7790] Revert changes in DFSPathSelector and UtilHelpers.readConfig [hudi]
yihua opened a new pull request, #11293: URL: https://github.com/apache/hudi/pull/11293 ### Change Logs This PR reverts the changes in `DFSPathSelector`, subclasses, and `UtilHelpers.readConfig` API so they still use Hadoop classes, given they are used in Spark and utilities classes, to avoid behavior change and breaking API change. ### Impact As above, makes upgrade easier. ### Risk level none ### Documentation Update none ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Change Logs and Impact were stated clearly - [ ] Adequate tests were added if applicable - [ ] CI passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7776] Simplify HoodieStorage instance fetching [hudi]
yihua merged PR #11259: URL: https://github.com/apache/hudi/pull/11259 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7776][DNM][Testing] Simplify HoodieStorage instance fetching [hudi]
yihua closed pull request #11291: [HUDI-7776][DNM][Testing] Simplify HoodieStorage instance fetching URL: https://github.com/apache/hudi/pull/11291 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7776] Simplify HoodieStorage instance fetching [hudi]
hudi-bot commented on PR #11292: URL: https://github.com/apache/hudi/pull/11292#issuecomment-2130519492 ## CI report: * f9369ce9e7345b01f35abe4a3b0c03dc31a26bfd UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-7790) Revert changes in DFSPathSelector and UtilHelpers.readConfig
[ https://issues.apache.org/jira/browse/HUDI-7790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7790: Description: This is to avoid behavior changes in > Revert changes in DFSPathSelector and UtilHelpers.readConfig > > > Key: HUDI-7790 > URL: https://issues.apache.org/jira/browse/HUDI-7790 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Assignee: Ethan Guo >Priority: Major > Fix For: 0.15.0, 1.0.0 > > > This is to avoid behavior changes in -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HUDI-7790) Revert changes in DFSPathSelector and UtilHelpers.readConfig
[ https://issues.apache.org/jira/browse/HUDI-7790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7790: Description: This is to avoid behavior changes in DFSPathSelector and keep the UtilHelpers.readConfig API the same as before. was:This is to avoid behavior changes in DFSPathSelector and keep the DFSPathSelector > Revert changes in DFSPathSelector and UtilHelpers.readConfig > > > Key: HUDI-7790 > URL: https://issues.apache.org/jira/browse/HUDI-7790 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Assignee: Ethan Guo >Priority: Major > Fix For: 0.15.0, 1.0.0 > > > This is to avoid behavior changes in DFSPathSelector and keep the > UtilHelpers.readConfig API the same as before. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HUDI-7790) Revert changes in DFSPathSelector and UtilHelpers.readConfig
[ https://issues.apache.org/jira/browse/HUDI-7790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7790: Description: This is to avoid behavior changes in DFSPathSelector and keep the (was: This is to avoid behavior changes in ) > Revert changes in DFSPathSelector and UtilHelpers.readConfig > > > Key: HUDI-7790 > URL: https://issues.apache.org/jira/browse/HUDI-7790 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Assignee: Ethan Guo >Priority: Major > Fix For: 0.15.0, 1.0.0 > > > This is to avoid behavior changes in DFSPathSelector and keep the -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HUDI-7790) Revert changes in DFSPathSelector and UtilHelpers.readConfig
[ https://issues.apache.org/jira/browse/HUDI-7790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7790: Description: This is to avoid behavior changes in DFSPathSelector and keep the DFSPathSelector (was: This is to avoid behavior changes in DFSPathSelector and keep the ) > Revert changes in DFSPathSelector and UtilHelpers.readConfig > > > Key: HUDI-7790 > URL: https://issues.apache.org/jira/browse/HUDI-7790 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Assignee: Ethan Guo >Priority: Major > Fix For: 0.15.0, 1.0.0 > > > This is to avoid behavior changes in DFSPathSelector and keep the > DFSPathSelector -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HUDI-7790) Revert changes in DFSPathSelector and UtilHelpers.readConfig
Ethan Guo created HUDI-7790: --- Summary: Revert changes in DFSPathSelector and UtilHelpers.readConfig Key: HUDI-7790 URL: https://issues.apache.org/jira/browse/HUDI-7790 Project: Apache Hudi Issue Type: Improvement Reporter: Ethan Guo -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HUDI-7790) Revert changes in DFSPathSelector and UtilHelpers.readConfig
[ https://issues.apache.org/jira/browse/HUDI-7790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo reassigned HUDI-7790: --- Assignee: Ethan Guo > Revert changes in DFSPathSelector and UtilHelpers.readConfig > > > Key: HUDI-7790 > URL: https://issues.apache.org/jira/browse/HUDI-7790 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Assignee: Ethan Guo >Priority: Major > Fix For: 0.15.0, 1.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HUDI-7790) Revert changes in DFSPathSelector and UtilHelpers.readConfig
[ https://issues.apache.org/jira/browse/HUDI-7790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7790: Fix Version/s: 0.15.0 1.0.0 > Revert changes in DFSPathSelector and UtilHelpers.readConfig > > > Key: HUDI-7790 > URL: https://issues.apache.org/jira/browse/HUDI-7790 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Priority: Major > Fix For: 0.15.0, 1.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [HUDI-7776] Simplify HoodieStorage instance fetching [hudi]
yihua opened a new pull request, #11292: URL: https://github.com/apache/hudi/pull/11292 ### Change Logs This PR simplifies `HoodieStorage` instance fetching to pass down the `HoodieStorage` instance from the meta client as much as possible, instead of using reflection, which may not work with a given file system instance. ### Impact As above. ### Risk level low ### Documentation Update none ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Change Logs and Impact were stated clearly - [ ] Adequate tests were added if applicable - [ ] CI passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7776] Simplify HoodieStorage instance fetching [hudi]
yihua commented on PR #11259: URL: https://github.com/apache/hudi/pull/11259#issuecomment-2130483050 Azure CI is green on another PR with the same changes (this PR could not trigger Azure CI somehow). https://github.com/apache/hudi/assets/2497195/4573b294-9cf9-47c8-843e-ce6cf1745e7e;> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7774] Add Avro Logical type support for Merciful Java convertor [hudi]
Davis-Zhang-Onehouse commented on code in PR #11265:
URL: https://github.com/apache/hudi/pull/11265#discussion_r1614082202
##
hudi-common/src/main/java/org/apache/hudi/avro/AvroLogicalTypeEnum.java:
##
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.avro;
+
+public enum AvroLogicalTypeEnum {
Review Comment:
addressed offline, Ethan is on the same page that we create our own Enum as
Avro does not have such thing published for consumption.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7776][DNM][Testing] Simplify HoodieStorage instance fetching [hudi]
hudi-bot commented on PR #11291: URL: https://github.com/apache/hudi/pull/11291#issuecomment-2130391866 ## CI report: * 75d67c7b784f96096892e720298c002d34637b75 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
