Re: [PR] [HUDI-7528] Fixing RowCustomColumnsSortPartitioner to use repartition instead of coalesce [hudi]
hudi-bot commented on PR #10909: URL: https://github.com/apache/hudi/pull/10909#issuecomment-2092181639 ## CI report: * 78efc7ca1cc033e445086b925cae48204d214871 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23642) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7508] Avoid collecting records in HoodieStreamerUtils.createHoodieRecords and JsonKafkaSource mapPartitions [hudi]
hudi-bot commented on PR #10872: URL: https://github.com/apache/hudi/pull/10872#issuecomment-2092180728 ## CI report: * ac7713c64afa1d2406463c8563a065362c95ecda Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23640) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7701] Metadata table initailization with pending instants [hudi]
hudi-bot commented on PR #11137: URL: https://github.com/apache/hudi/pull/11137#issuecomment-2092096715 ## CI report: * fe7584c435e0ba03e9176cf5e7cc331d9a0052d7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23614) * a668de4b47df64e2d09b8c1bd0a172271c41a7e3 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23644) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7707] Enable bundle validation on Java 8 and 11 [hudi]
hudi-bot commented on PR #11142: URL: https://github.com/apache/hudi/pull/11142#issuecomment-2092096740 ## CI report: * fd5383cabb77ad3afc075ee1545e65c7e0613855 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23638) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7701] Metadata table initailization with pending instants [hudi]
hudi-bot commented on PR #11137: URL: https://github.com/apache/hudi/pull/11137#issuecomment-2092086614 ## CI report: * fe7584c435e0ba03e9176cf5e7cc331d9a0052d7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23614) * a668de4b47df64e2d09b8c1bd0a172271c41a7e3 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7707] Enable bundle validation on Java 8 and 11 [hudi]
hudi-bot commented on PR #11142: URL: https://github.com/apache/hudi/pull/11142#issuecomment-2092079871 ## CI report: * 14896f28dc869895f9f7897354e2807c52140607 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23636) * fd5383cabb77ad3afc075ee1545e65c7e0613855 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23638) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7523] Add HOODIE_SPARK_DATASOURCE_OPTIONS to be used in HoodieIncrSource [hudi]
hudi-bot commented on PR #10900: URL: https://github.com/apache/hudi/pull/10900#issuecomment-2092079595 ## CI report: * 5fefa9e02c016d50b2f2b1fda2c9c89f2df7d620 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23641) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(hudi) branch master updated: [HUDI-7686] Add tests on the util methods for type cast of configuration instances (#11121)
This is an automated email from the ASF dual-hosted git repository.
yihua pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/master by this push:
new feb82e61a06 [HUDI-7686] Add tests on the util methods for type cast of
configuration instances (#11121)
feb82e61a06 is described below
commit feb82e61a06024ddf1efdcad184f2da6e705062a
Author: Y Ethan Guo
AuthorDate: Thu May 2 20:55:00 2024 -0700
[HUDI-7686] Add tests on the util methods for type cast of configuration
instances (#11121)
---
.../io/storage/BaseTestStorageConfiguration.java | 29 ++
1 file changed, 24 insertions(+), 5 deletions(-)
diff --git
a/hudi-io/src/test/java/org/apache/hudi/io/storage/BaseTestStorageConfiguration.java
b/hudi-io/src/test/java/org/apache/hudi/io/storage/BaseTestStorageConfiguration.java
index 1d6a3d338e4..3bc575e3dff 100644
---
a/hudi-io/src/test/java/org/apache/hudi/io/storage/BaseTestStorageConfiguration.java
+++
b/hudi-io/src/test/java/org/apache/hudi/io/storage/BaseTestStorageConfiguration.java
@@ -37,6 +37,7 @@ import static org.junit.jupiter.api.Assertions.assertFalse;
import static org.junit.jupiter.api.Assertions.assertNotNull;
import static org.junit.jupiter.api.Assertions.assertNotSame;
import static org.junit.jupiter.api.Assertions.assertSame;
+import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
/**
@@ -71,13 +72,31 @@ public abstract class BaseTestStorageConfiguration {
@Test
public void testConstructorNewInstanceUnwrapCopy() {
-T conf = getConf(EMPTY_MAP);
+T conf = getConf(prepareConfigs());
StorageConfiguration storageConf = getStorageConfiguration(conf);
StorageConfiguration newStorageConf = storageConf.newInstance();
-assertNotSame(storageConf, newStorageConf);
-assertNotSame(storageConf.unwrap(), newStorageConf.unwrap());
-assertSame(storageConf.unwrap(), storageConf.unwrap());
-assertNotSame(storageConf.unwrap(), storageConf.unwrapCopy());
+Class unwrapperConfClass = storageConf.unwrap().getClass();
+assertNotSame(storageConf, newStorageConf,
+"storageConf.newInstance() should return a different
StorageConfiguration instance.");
+validateConfigs(newStorageConf);
+assertNotSame(storageConf.unwrap(), newStorageConf.unwrap(),
+"storageConf.newInstance() should contain a new copy of the underlying
configuration instance.");
+assertSame(storageConf.unwrap(), storageConf.unwrap(),
+"storageConf.unwrap() should return the same underlying configuration
instance.");
+assertSame(storageConf.unwrap(), storageConf.unwrapAs(unwrapperConfClass),
+"storageConf.unwrapAs(unwrapperConfClass) should return the same
underlying configuration instance.");
+assertNotSame(storageConf.unwrap(), storageConf.unwrapCopy(),
+"storageConf.unwrapCopy() should return a new copy of the underlying
configuration instance.");
+validateConfigs(getStorageConfiguration(storageConf.unwrapCopy()));
+assertNotSame(storageConf.unwrap(),
storageConf.unwrapCopyAs(unwrapperConfClass),
+"storageConf.unwrapCopyAs(unwrapperConfClass) should return a new copy
of the underlying configuration instance.");
+validateConfigs(getStorageConfiguration((T)
storageConf.unwrapCopyAs(unwrapperConfClass)));
+assertThrows(
+IllegalArgumentException.class,
+() -> storageConf.unwrapAs(Integer.class));
+assertThrows(
+IllegalArgumentException.class,
+() -> storageConf.unwrapCopyAs(Integer.class));
}
@Test
Re: [PR] [HUDI-7686] Add tests on the util methods for type cast of configuration instances [hudi]
yihua merged PR #11121: URL: https://github.com/apache/hudi/pull/11121 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]
ksoullpwk commented on PR #11130: URL: https://github.com/apache/hudi/pull/11130#issuecomment-2092071157 Do we have a plan to apply the Scala 2.13 support for older Spark versions? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-7708) Support cleaning archived commits
[ https://issues.apache.org/jira/browse/HUDI-7708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-7708: - Description: currently archived commits in 0.x can take up a lot storage space and users are not sure if manually delete cause any issue. we would develop some mechanism to help optimize the storage for archived commits. related issues [https://github.com/apache/hudi/issues/7246] [https://github.com/apache/hudi/issues/7734] > Support cleaning archived commits > - > > Key: HUDI-7708 > URL: https://issues.apache.org/jira/browse/HUDI-7708 > Project: Apache Hudi > Issue Type: Improvement > Components: archiving, table-service >Reporter: Raymond Xu >Priority: Major > Fix For: 0.15.0 > > > currently archived commits in 0.x can take up a lot storage space and users > are not sure if manually delete cause any issue. we would develop some > mechanism to help optimize the storage for archived commits. > > related issues > [https://github.com/apache/hudi/issues/7246] > [https://github.com/apache/hudi/issues/7734] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HUDI-7708) Support cleaning archived commits
Raymond Xu created HUDI-7708: Summary: Support cleaning archived commits Key: HUDI-7708 URL: https://issues.apache.org/jira/browse/HUDI-7708 Project: Apache Hudi Issue Type: Improvement Components: archiving, table-service Reporter: Raymond Xu Fix For: 0.15.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [HUDI-7528] Fixing RowCustomColumnsSortPartitioner to use repartition instead of coalesce [hudi]
hudi-bot commented on PR #10909: URL: https://github.com/apache/hudi/pull/10909#issuecomment-2092056966 ## CI report: * b5ebcf8de8abc367918e5ab570be4bcd52b33208 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22990) * 78efc7ca1cc033e445086b925cae48204d214871 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23642) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7523] Add HOODIE_SPARK_DATASOURCE_OPTIONS to be used in HoodieIncrSource [hudi]
hudi-bot commented on PR #10900: URL: https://github.com/apache/hudi/pull/10900#issuecomment-2092056931 ## CI report: * 6800d009ebd79b21a6134aab7db352ab5d5d1ae3 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22975) * 5fefa9e02c016d50b2f2b1fda2c9c89f2df7d620 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23641) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7508] Avoid collecting records in HoodieStreamerUtils.createHoodieRecords and JsonKafkaSource mapPartitions [hudi]
hudi-bot commented on PR #10872: URL: https://github.com/apache/hudi/pull/10872#issuecomment-2092056890 ## CI report: * 629e91bc0267c0728b98326eb84072965c600205 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22928) * ac7713c64afa1d2406463c8563a065362c95ecda Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23640) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7429] Fixing average record size estimation for delta commits [hudi]
hudi-bot commented on PR #10763: URL: https://github.com/apache/hudi/pull/10763#issuecomment-2092056831 ## CI report: * 34711f0b4fd724a5a6631b4fdacd90acebe53ca1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23639) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7707] Enable bundle validation on Java 8 and 11 [hudi]
hudi-bot commented on PR #11142: URL: https://github.com/apache/hudi/pull/11142#issuecomment-2092052949 ## CI report: * d6d48bad055f3f2b41a974f69031a49013c7175a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23634) * 14896f28dc869895f9f7897354e2807c52140607 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23636) * fd5383cabb77ad3afc075ee1545e65c7e0613855 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23638) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7528] Fixing RowCustomColumnsSortPartitioner to use repartition instead of coalesce [hudi]
hudi-bot commented on PR #10909: URL: https://github.com/apache/hudi/pull/10909#issuecomment-2092052645 ## CI report: * b5ebcf8de8abc367918e5ab570be4bcd52b33208 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22990) * 78efc7ca1cc033e445086b925cae48204d214871 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7523] Add HOODIE_SPARK_DATASOURCE_OPTIONS to be used in HoodieIncrSource [hudi]
hudi-bot commented on PR #10900: URL: https://github.com/apache/hudi/pull/10900#issuecomment-2092052606 ## CI report: * 6800d009ebd79b21a6134aab7db352ab5d5d1ae3 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22975) * 5fefa9e02c016d50b2f2b1fda2c9c89f2df7d620 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7429] Fixing average record size estimation for delta commits [hudi]
hudi-bot commented on PR #10763: URL: https://github.com/apache/hudi/pull/10763#issuecomment-2092052479 ## CI report: * 95411f507afa43c6e5bf95e8bf1f87bbc03beb49 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22702) * 34711f0b4fd724a5a6631b4fdacd90acebe53ca1 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7508] Avoid collecting records in HoodieStreamerUtils.createHoodieRecords and JsonKafkaSource mapPartitions [hudi]
hudi-bot commented on PR #10872: URL: https://github.com/apache/hudi/pull/10872#issuecomment-2092052549 ## CI report: * 629e91bc0267c0728b98326eb84072965c600205 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=22928) * ac7713c64afa1d2406463c8563a065362c95ecda UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7707] Enable bundle validation on Java 8 and 11 [hudi]
hudi-bot commented on PR #11142: URL: https://github.com/apache/hudi/pull/11142#issuecomment-2092049159 ## CI report: * d6d48bad055f3f2b41a974f69031a49013c7175a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23634) * 14896f28dc869895f9f7897354e2807c52140607 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23636) * fd5383cabb77ad3afc075ee1545e65c7e0613855 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7686] Add tests on the util methods for type cast of configuration instances [hudi]
hudi-bot commented on PR #11121: URL: https://github.com/apache/hudi/pull/11121#issuecomment-2092049104 ## CI report: * 9ebc7b514587fec7a1d2b9ca559d9cf655dbb6b0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23633) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7532] Include only compaction instants for lastCompaction in getDeltaCommitsSinceLatestCompaction [hudi]
yihua commented on PR #10915: URL: https://github.com/apache/hudi/pull/10915#issuecomment-2092034933 @nsivabalan could you rebase the PR on the latest master and address the review comments? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7501] Use source profile for S3 and GCS sources [hudi]
yihua commented on PR #10861: URL: https://github.com/apache/hudi/pull/10861#issuecomment-2092032092 @vinishjail97 could you rebase this PR on the latest master? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7707] Enable bundle validation on Java 8 and 11 [hudi]
hudi-bot commented on PR #11142: URL: https://github.com/apache/hudi/pull/11142#issuecomment-2092027530 ## CI report: * d6d48bad055f3f2b41a974f69031a49013c7175a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23634) * 14896f28dc869895f9f7897354e2807c52140607 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23636) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-7704) Unify test client storage classes with duplicate code
[ https://issues.apache.org/jira/browse/HUDI-7704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vova Kolmakov updated HUDI-7704: Status: In Progress (was: Open) > Unify test client storage classes with duplicate code > -- > > Key: HUDI-7704 > URL: https://issues.apache.org/jira/browse/HUDI-7704 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Jonathan Vexler >Assignee: Vova Kolmakov >Priority: Major > > TestHoodieClientOnCopyOnWriteStorage > TestHoodieJavaClientOnCopyOnWriteStorage > have a bunch of duplicate code -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HUDI-7540) Check for gaps on storing inserts on log files
[ https://issues.apache.org/jira/browse/HUDI-7540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar resolved HUDI-7540. -- > Check for gaps on storing inserts on log files > -- > > Key: HUDI-7540 > URL: https://issues.apache.org/jira/browse/HUDI-7540 > Project: Apache Hudi > Issue Type: Improvement > Components: storage-management >Reporter: Vinoth Chandar >Assignee: Vinoth Chandar >Priority: Major > Fix For: 1.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HUDI-7234) Handle both inserts and updates in log blocks for partial updates
[ https://issues.apache.org/jira/browse/HUDI-7234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-7234: - Description: Inserts can be written to log blocks, e.g., Flink. We need to handle such case for partial updates i.e mix of inserts and partial updates to the same data block. (was: Inserts can be written to log blocks, e.g., Flink. We need to handle such case for partial updates.) > Handle both inserts and updates in log blocks for partial updates > - > > Key: HUDI-7234 > URL: https://issues.apache.org/jira/browse/HUDI-7234 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Assignee: Vinoth Chandar >Priority: Blocker > Fix For: 1.0.0 > > > Inserts can be written to log blocks, e.g., Flink. We need to handle such > case for partial updates i.e mix of inserts and partial updates to the same > data block. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (HUDI-7540) Check for gaps on storing inserts on log files
[ https://issues.apache.org/jira/browse/HUDI-7540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar closed HUDI-7540. Resolution: Invalid > Check for gaps on storing inserts on log files > -- > > Key: HUDI-7540 > URL: https://issues.apache.org/jira/browse/HUDI-7540 > Project: Apache Hudi > Issue Type: Improvement > Components: storage-management >Reporter: Vinoth Chandar >Assignee: Vinoth Chandar >Priority: Major > Fix For: 1.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HUDI-7229) Enable partial updates for CDC work payload
[ https://issues.apache.org/jira/browse/HUDI-7229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-7229: - Description: OLTP workloads on upstream databases, often update/delete/insert different columns in the table on each operation. Currently, Hudi can only supporting partial updates in cases where the same columns are being mutated in a given write to Hudi (e.g Spark SQL ETLs with MIT or Update statements). Here, we explore what it takes to support a smarter storage format, that can only encode the changed columns into log along with the different implementations. h2. Goals # Enable partial update functionality for all existing and potential future CDC workloads without huge modification or duplication. # Performance parity with current full-record updates or partial updates across the same set of columns # Exhibit reduction in storage costs, by only storing the changed columns. # Should also result in computation cost reductions by scanning/processing less data # Should not affect the scalability of the existing system ingestion system. The number of files generated for partial update should not increase dramatically. was:DMS, Debezium, etc. > Enable partial updates for CDC work payload > --- > > Key: HUDI-7229 > URL: https://issues.apache.org/jira/browse/HUDI-7229 > Project: Apache Hudi > Issue Type: Task >Reporter: Lin Liu >Assignee: Vinoth Chandar >Priority: Major > Labels: pull-request-available > Fix For: 1.1.0 > > > OLTP workloads on upstream databases, often update/delete/insert different > columns in the table on each operation. Currently, Hudi can only supporting > partial updates in cases where the same columns are being mutated in a given > write to Hudi (e.g Spark SQL ETLs with MIT or Update statements). Here, we > explore what it takes to support a smarter storage format, that can only > encode the changed columns into log along with the different implementations. > h2. Goals > # Enable partial update functionality for all existing and potential future > CDC workloads without huge modification or duplication. > # Performance parity with current full-record updates or partial updates > across the same set of columns > # Exhibit reduction in storage costs, by only storing the changed columns. > # Should also result in computation cost reductions by scanning/processing > less data > # Should not affect the scalability of the existing system ingestion system. > The number of files generated for partial update should not increase > dramatically. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HUDI-7229) Enable partial updates for CDC work payload
[ https://issues.apache.org/jira/browse/HUDI-7229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843110#comment-17843110 ] Vinoth Chandar commented on HUDI-7229: -- Punting this to 1.1 # [1.1] Implement support on top of data blocks. ## we need to pass change columns information and operation all the way to write handles, using a field in HoodieRecord ## ... # [1.1] Implement support on top of cdc data blocks. ## we can track similar bitmaps for cdc data blocks as well ## we need to extend the new file group reader to also merge base and cdc blocks. (not just base and data blocks). > Enable partial updates for CDC work payload > --- > > Key: HUDI-7229 > URL: https://issues.apache.org/jira/browse/HUDI-7229 > Project: Apache Hudi > Issue Type: Task >Reporter: Lin Liu >Assignee: Vinoth Chandar >Priority: Major > Labels: pull-request-available > Fix For: 1.1.0 > > > OLTP workloads on upstream databases, often update/delete/insert different > columns in the table on each operation. Currently, Hudi can only supporting > partial updates in cases where the same columns are being mutated in a given > write to Hudi (e.g Spark SQL ETLs with MIT or Update statements). Here, we > explore what it takes to support a smarter storage format, that can only > encode the changed columns into log along with the different implementations. > h2. Goals > # Enable partial update functionality for all existing and potential future > CDC workloads without huge modification or duplication. > # Performance parity with current full-record updates or partial updates > across the same set of columns > # Exhibit reduction in storage costs, by only storing the changed columns. > # Should also result in computation cost reductions by scanning/processing > less data > # Should not affect the scalability of the existing system ingestion system. > The number of files generated for partial update should not increase > dramatically. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [MINOR] remove unnecessary lines from java test [hudi]
danny0405 merged PR #11139: URL: https://github.com/apache/hudi/pull/11139 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(hudi) branch master updated: [MINOR] remove unnecessary lines from java test (#11139)
This is an automated email from the ASF dual-hosted git repository.
danny0405 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/master by this push:
new a0ea78fdd8d [MINOR] remove unnecessary lines from java test (#11139)
a0ea78fdd8d is described below
commit a0ea78fdd8df1a2be59a28a7b62427459996d710
Author: Jon Vexler
AuthorDate: Thu May 2 22:13:43 2024 -0400
[MINOR] remove unnecessary lines from java test (#11139)
Co-authored-by: Jonathan Vexler <=>
---
.../client/functional/TestHoodieJavaClientOnCopyOnWriteStorage.java | 4
1 file changed, 4 deletions(-)
diff --git
a/hudi-client/hudi-java-client/src/test/java/org/apache/hudi/client/functional/TestHoodieJavaClientOnCopyOnWriteStorage.java
b/hudi-client/hudi-java-client/src/test/java/org/apache/hudi/client/functional/TestHoodieJavaClientOnCopyOnWriteStorage.java
index 5193b859908..8e9cbce0c92 100644
---
a/hudi-client/hudi-java-client/src/test/java/org/apache/hudi/client/functional/TestHoodieJavaClientOnCopyOnWriteStorage.java
+++
b/hudi-client/hudi-java-client/src/test/java/org/apache/hudi/client/functional/TestHoodieJavaClientOnCopyOnWriteStorage.java
@@ -578,10 +578,6 @@ public class TestHoodieJavaClientOnCopyOnWriteStorage
extends HoodieJavaClientTe
partitionPath, FSUtils.getFileId(baseFilePath.getName()), baseFile,
new JavaTaskContextSupplier(),
config.populateMetaFields() ? Option.empty() :
Option.of((BaseKeyGenerator)
HoodieAvroKeyGeneratorFactory.createKeyGenerator(new
TypedProperties(config.getProps();
- WriteStatus writeStatus = new WriteStatus(false, 0.0);
- writeStatus.setStat(new HoodieWriteStat());
- writeStatus.getStat().setNumWrites(0);
- handle.performMergeDataValidationCheck(writeStatus);
fail("The above line should have thrown an exception");
} catch (HoodieUpsertException e2) {
// expected
[jira] [Updated] (HUDI-7229) Enable partial updates for CDC work payload
[ https://issues.apache.org/jira/browse/HUDI-7229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar updated HUDI-7229: - Fix Version/s: 1.1.0 (was: 1.0.0) > Enable partial updates for CDC work payload > --- > > Key: HUDI-7229 > URL: https://issues.apache.org/jira/browse/HUDI-7229 > Project: Apache Hudi > Issue Type: Task >Reporter: Lin Liu >Assignee: Vinoth Chandar >Priority: Major > Labels: pull-request-available > Fix For: 1.1.0 > > > DMS, Debezium, etc. -- This message was sent by Atlassian Jira (v8.20.10#820010)
(hudi) branch master updated (3930119544d -> aff11ac2c2f)
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from 3930119544d [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration (#11130) add aff11ac2c2f [HUDI-7688] Stop retry inflate if encounter InterruptedIOException (#11125) No new revisions were added by this update. Summary of changes: .../java/org/apache/hudi/common/table/log/block/HoodieLogBlock.java | 5 + 1 file changed, 5 insertions(+)
Re: [PR] [HUDI-7688] Stop retry inflate if encounter InterruptedIOException [hudi]
danny0405 merged PR #11125: URL: https://github.com/apache/hudi/pull/11125 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7707] Enable bundle validation on Java 8 and 11 [hudi]
hudi-bot commented on PR #11142: URL: https://github.com/apache/hudi/pull/11142#issuecomment-2092023164 ## CI report: * d6d48bad055f3f2b41a974f69031a49013c7175a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23634) * 14896f28dc869895f9f7897354e2807c52140607 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7429] Fixing average record size estimation for delta commits [hudi]
yihua commented on code in PR #10763:
URL: https://github.com/apache/hudi/pull/10763#discussion_r1588601099
##
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/table/action/commit/AverageRecordSizeUtils.java:
##
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.table.action.commit;
+
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.config.HoodieWriteConfig;
+
+import org.apache.hadoop.fs.Path;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Iterator;
+import java.util.concurrent.atomic.AtomicLong;
+
+import static
org.apache.hudi.common.table.timeline.HoodieTimeline.COMMIT_ACTION;
+import static
org.apache.hudi.common.table.timeline.HoodieTimeline.DELTA_COMMIT_ACTION;
+import static
org.apache.hudi.common.table.timeline.HoodieTimeline.REPLACE_COMMIT_ACTION;
+
+/**
+ * Util class to assist with fetching average record size.
+ */
+public class AverageRecordSizeUtils {
+ private static final Logger LOG =
LoggerFactory.getLogger(AverageRecordSizeUtils.class);
+
+ /**
+ * Obtains the average record size based on records written during previous
commits. Used for estimating how many
+ * records pack into one file.
+ */
+ static long averageBytesPerRecord(HoodieTimeline commitTimeline,
HoodieWriteConfig hoodieWriteConfig) {
+long avgSize = hoodieWriteConfig.getCopyOnWriteRecordSizeEstimate();
+long fileSizeThreshold = (long)
(hoodieWriteConfig.getRecordSizeEstimationThreshold() *
hoodieWriteConfig.getParquetSmallFileLimit());
+try {
Review Comment:
@nsivabalan have you addressed the comment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (HUDI-7671) Make Hudi timeline backward compatible
[
https://issues.apache.org/jira/browse/HUDI-7671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843109#comment-17843109
]
Vinoth Chandar commented on HUDI-7671:
--
balaji - this may be a dupe.
> Make Hudi timeline backward compatible
> --
>
> Key: HUDI-7671
> URL: https://issues.apache.org/jira/browse/HUDI-7671
> Project: Apache Hudi
> Issue Type: Improvement
> Components: core
>Reporter: Danny Chen
>Assignee: Balaji Varadarajan
>Priority: Major
> Labels: compatibility
> Fix For: 1.0.0
>
>
> Since release 1.x, the timeline metadata file name is changed to include the
> completion time, we need to keep compatibility for 0.x branches/releases.
> 0.x meta file name pattern: ${instant_time}.action[.state]
> 1.x meta file name pattern: ${instant_time}_${completion_time}.action[.state].
> In 1.x release, while decipher the Hudi instant from the metadata files, if
> there is no completion time, uses the file modification time as the
> completion time instead.
> The modification time follows the OCC concurrency control semantics if the
> files were not moved around.
> Caution that if the table is a MOR table and the files got moved in history
> from old folder to the current folder, the reader view may represent wong
> result set because the completion time are completely the same for all the
> alive instants.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HUDI-7671) Make Hudi timeline backward compatible
[
https://issues.apache.org/jira/browse/HUDI-7671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinoth Chandar updated HUDI-7671:
-
Epic Link: HUDI-6242
> Make Hudi timeline backward compatible
> --
>
> Key: HUDI-7671
> URL: https://issues.apache.org/jira/browse/HUDI-7671
> Project: Apache Hudi
> Issue Type: Improvement
> Components: core
>Reporter: Danny Chen
>Assignee: Danny Chen
>Priority: Major
> Labels: compatibility
> Fix For: 1.0.0
>
>
> Since release 1.x, the timeline metadata file name is changed to include the
> completion time, we need to keep compatibility for 0.x branches/releases.
> 0.x meta file name pattern: ${instant_time}.action[.state]
> 1.x meta file name pattern: ${instant_time}_${completion_time}.action[.state].
> In 1.x release, while decipher the Hudi instant from the metadata files, if
> there is no completion time, uses the file modification time as the
> completion time instead.
> The modification time follows the OCC concurrency control semantics if the
> files were not moved around.
> Caution that if the table is a MOR table and the files got moved in history
> from old folder to the current folder, the reader view may represent wong
> result set because the completion time are completely the same for all the
> alive instants.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HUDI-7671) Make Hudi timeline backward compatible
[
https://issues.apache.org/jira/browse/HUDI-7671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinoth Chandar reassigned HUDI-7671:
Assignee: Balaji Varadarajan (was: Danny Chen)
> Make Hudi timeline backward compatible
> --
>
> Key: HUDI-7671
> URL: https://issues.apache.org/jira/browse/HUDI-7671
> Project: Apache Hudi
> Issue Type: Improvement
> Components: core
>Reporter: Danny Chen
>Assignee: Balaji Varadarajan
>Priority: Major
> Labels: compatibility
> Fix For: 1.0.0
>
>
> Since release 1.x, the timeline metadata file name is changed to include the
> completion time, we need to keep compatibility for 0.x branches/releases.
> 0.x meta file name pattern: ${instant_time}.action[.state]
> 1.x meta file name pattern: ${instant_time}_${completion_time}.action[.state].
> In 1.x release, while decipher the Hudi instant from the metadata files, if
> there is no completion time, uses the file modification time as the
> completion time instead.
> The modification time follows the OCC concurrency control semantics if the
> files were not moved around.
> Caution that if the table is a MOR table and the files got moved in history
> from old folder to the current folder, the reader view may represent wong
> result set because the completion time are completely the same for all the
> alive instants.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HUDI-7678) Finalize the Merger APIs and make a plan for moving over all existing built-in, custom payloads.
[
https://issues.apache.org/jira/browse/HUDI-7678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinoth Chandar updated HUDI-7678:
-
Description:
With the move towards making partial updates a first class citizen, that does
not need any special payloads/merges, we need to move the CDC payloads to all
be transformers in Hudi Streamer and SQL write path. Along with migration
instructions to users.
# partial update has been implemented for Spark SQL source as follows:
## Configuration \{{ hoodie.write.partial.update.schema }} is used for partial
update.
## {{ExpressionPayload}} creates the writer schema based on the configuration.
## {{HoodieAppendHandle}} creates the log file based on the confgiuration and
the corresponding partial schema.
## Currently this handle assumes these records are all update records.
## We need to understand if ExpressionPayload/SQL Merger is needed to going
forward.
# For DeltaStreamer, our goal is to remove all silo CDC payloads, e.g.,
Debezium or AWSDMS, and to provide CDC data as {{InternalRow}} type. Therefore,
## The {{transformer}} in DeltaStreamer prepares the data according to the
types of the sources.
## Initially, its okay to just support full row updates/deletes/...
# Audit all of them should properly combine I/U/D into data and delete blocks,
such that U after D, D after U scenarios are handled as expected.
was:
With the move towards making partial updates a first class citizen, that does
not need any special payloads/merges, we need to move the CDC payloads to all
be transformers in Hudi Streamer and SQL write path. Along with migration
instructions to users.
# partial update has been implemented for Spark SQL source as follows:
## Configuration {{ hoodie.write.partial.update.schema }} is used for partial
update.
## {{ExpressionPayload}} creates the writer schema based on the configuration.
## {{HoodieAppendHandle}} creates the log file based on the confgiuration and
the corresponding partial schema.
## Currently this handle assumes these records are all update records.
## We need to understand if ExpressionPayload/SQL Merger is needed to going
forward.
# For DeltaStreamer, our goal is to remove all silo CDC payloads, e.g.,
Debezium or AWSDMS, and to provide CDC data as {{InternalRow}} type. Therefore,
## The {{transformer}} in DeltaStreamer prepares the data according to the
types of the sources.
## Initially, its okay to just support full row updates/deletes/...
> Finalize the Merger APIs and make a plan for moving over all existing
> built-in, custom payloads.
>
>
> Key: HUDI-7678
> URL: https://issues.apache.org/jira/browse/HUDI-7678
> Project: Apache Hudi
> Issue Type: Task
>Reporter: Vinoth Chandar
>Assignee: Vinoth Chandar
>Priority: Major
> Fix For: 1.0.0
>
>
> With the move towards making partial updates a first class citizen, that does
> not need any special payloads/merges, we need to move the CDC payloads to all
> be transformers in Hudi Streamer and SQL write path. Along with migration
> instructions to users.
> # partial update has been implemented for Spark SQL source as follows:
> ## Configuration \{{ hoodie.write.partial.update.schema }} is used for
> partial update.
> ## {{ExpressionPayload}} creates the writer schema based on the
> configuration.
> ## {{HoodieAppendHandle}} creates the log file based on the confgiuration
> and the corresponding partial schema.
> ## Currently this handle assumes these records are all update records.
> ## We need to understand if ExpressionPayload/SQL Merger is needed to going
> forward.
> # For DeltaStreamer, our goal is to remove all silo CDC payloads, e.g.,
> Debezium or AWSDMS, and to provide CDC data as {{InternalRow}} type.
> Therefore,
> ## The {{transformer}} in DeltaStreamer prepares the data according to the
> types of the sources.
> ## Initially, its okay to just support full row updates/deletes/...
> # Audit all of them should properly combine I/U/D into data and delete
> blocks, such that U after D, D after U scenarios are handled as expected.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [HUDI-7707] Enable bundle validation on Java 8 and 11 [hudi]
hudi-bot commented on PR #11142: URL: https://github.com/apache/hudi/pull/11142#issuecomment-2091995502 ## CI report: * d6d48bad055f3f2b41a974f69031a49013c7175a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23634) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7686] Add tests on the util methods for type cast of configuration instances [hudi]
hudi-bot commented on PR #11121: URL: https://github.com/apache/hudi/pull/11121#issuecomment-2091995443 ## CI report: * 6ada8df1f7a64c852ea031b634a160b2db92850b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23569) * 9ebc7b514587fec7a1d2b9ca559d9cf655dbb6b0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23633) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]
yihua merged PR #11130: URL: https://github.com/apache/hudi/pull/11130 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7707] Enable bundle validation on Java 8 and 11 [hudi]
hudi-bot commented on PR #11142: URL: https://github.com/apache/hudi/pull/11142#issuecomment-2091987890 ## CI report: * d6d48bad055f3f2b41a974f69031a49013c7175a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]
hudi-bot commented on PR #11130: URL: https://github.com/apache/hudi/pull/11130#issuecomment-2091987760 ## CI report: * edf2bf30a2ddbd48db9452f34b1ac716bd2ebe18 UNKNOWN * b1598f5861c2b90da91ad33dc360533728ef7163 UNKNOWN * 9a2450a1bb4454ddc2c86791ce112201f431627a UNKNOWN * 8105ef96648ad16ec61237a974dbed9e6a2d2c8f Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23632) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7686] Add tests on the util methods for type cast of configuration instances [hudi]
hudi-bot commented on PR #11121: URL: https://github.com/apache/hudi/pull/11121#issuecomment-2091987642 ## CI report: * 6ada8df1f7a64c852ea031b634a160b2db92850b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23569) * 9ebc7b514587fec7a1d2b9ca559d9cf655dbb6b0 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-7707) Enable bundle validation on Java 8 and 11
[ https://issues.apache.org/jira/browse/HUDI-7707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-7707: - Labels: pull-request-available (was: ) > Enable bundle validation on Java 8 and 11 > - > > Key: HUDI-7707 > URL: https://issues.apache.org/jira/browse/HUDI-7707 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Assignee: Ethan Guo >Priority: Major > Labels: pull-request-available > Fix For: 0.15.0, 1.0.0 > > Attachments: Screenshot 2024-05-02 at 17.41.02.png > > > Bundle validation with Java 8 and 11 are somehow skipped in GH CI. They > should be enabled. !Screenshot 2024-05-02 at > 17.41.02.png|width=905,height=325! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [HUDI-7707] Enable bundle validation on Java 8 and 11 [hudi]
yihua opened a new pull request, #11142: URL: https://github.com/apache/hudi/pull/11142 ### Change Logs Bundle validation with Java 8 and 11 are somehow skipped in GH CI. This PR reenables them by fixing the `bot.yml`. ### Impact Improves bundle validation coverage. ### Risk level none ### Documentation Update none ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Change Logs and Impact were stated clearly - [ ] Adequate tests were added if applicable - [ ] CI passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-7707) Enable bundle validation on Java 8 and 11
[ https://issues.apache.org/jira/browse/HUDI-7707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7707: Status: Patch Available (was: In Progress) > Enable bundle validation on Java 8 and 11 > - > > Key: HUDI-7707 > URL: https://issues.apache.org/jira/browse/HUDI-7707 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Assignee: Ethan Guo >Priority: Major > Fix For: 0.15.0, 1.0.0 > > Attachments: Screenshot 2024-05-02 at 17.41.02.png > > > Bundle validation with Java 8 and 11 are somehow skipped in GH CI. They > should be enabled. !Screenshot 2024-05-02 at > 17.41.02.png|width=905,height=325! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HUDI-7707) Enable bundle validation on Java 8 and 11
[ https://issues.apache.org/jira/browse/HUDI-7707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7707: Story Points: 0 > Enable bundle validation on Java 8 and 11 > - > > Key: HUDI-7707 > URL: https://issues.apache.org/jira/browse/HUDI-7707 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Assignee: Ethan Guo >Priority: Major > Fix For: 0.15.0, 1.0.0 > > Attachments: Screenshot 2024-05-02 at 17.41.02.png > > > Bundle validation with Java 8 and 11 are somehow skipped in GH CI. They > should be enabled. !Screenshot 2024-05-02 at > 17.41.02.png|width=905,height=325! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HUDI-7702) Remove unused method in ReflectUtil
[ https://issues.apache.org/jira/browse/HUDI-7702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7702: Story Points: 0.5 > Remove unused method in ReflectUtil > --- > > Key: HUDI-7702 > URL: https://issues.apache.org/jira/browse/HUDI-7702 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Assignee: Ethan Guo >Priority: Major > Labels: pull-request-available > Fix For: 0.15.0, 1.0.0 > > > ReflectUtil#createInsertInto is no longer used in the repo and causes issue > for Scala 2.13 support. We should remove the unused method. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HUDI-7707) Enable bundle validation on Java 8 and 11
[ https://issues.apache.org/jira/browse/HUDI-7707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7707: Status: In Progress (was: Open) > Enable bundle validation on Java 8 and 11 > - > > Key: HUDI-7707 > URL: https://issues.apache.org/jira/browse/HUDI-7707 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Assignee: Ethan Guo >Priority: Major > Fix For: 0.15.0, 1.0.0 > > Attachments: Screenshot 2024-05-02 at 17.41.02.png > > > Bundle validation with Java 8 and 11 are somehow skipped in GH CI. They > should be enabled. !Screenshot 2024-05-02 at > 17.41.02.png|width=905,height=325! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (HUDI-7702) Remove unused method in ReflectUtil
[ https://issues.apache.org/jira/browse/HUDI-7702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo closed HUDI-7702. --- Resolution: Fixed > Remove unused method in ReflectUtil > --- > > Key: HUDI-7702 > URL: https://issues.apache.org/jira/browse/HUDI-7702 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Assignee: Ethan Guo >Priority: Major > Labels: pull-request-available > Fix For: 0.15.0, 1.0.0 > > > ReflectUtil#createInsertInto is no longer used in the repo and causes issue > for Scala 2.13 support. We should remove the unused method. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HUDI-7707) Enable bundle validation on Java 8 and 11
[ https://issues.apache.org/jira/browse/HUDI-7707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7707: Story Points: 0.5 (was: 0) > Enable bundle validation on Java 8 and 11 > - > > Key: HUDI-7707 > URL: https://issues.apache.org/jira/browse/HUDI-7707 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Assignee: Ethan Guo >Priority: Major > Fix For: 0.15.0, 1.0.0 > > Attachments: Screenshot 2024-05-02 at 17.41.02.png > > > Bundle validation with Java 8 and 11 are somehow skipped in GH CI. They > should be enabled. !Screenshot 2024-05-02 at > 17.41.02.png|width=905,height=325! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (HUDI-7706) Improve validation in PARTITION_STATS index test
[ https://issues.apache.org/jira/browse/HUDI-7706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo closed HUDI-7706. --- Resolution: Fixed > Improve validation in PARTITION_STATS index test > > > Key: HUDI-7706 > URL: https://issues.apache.org/jira/browse/HUDI-7706 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Assignee: Ethan Guo >Priority: Major > Labels: pull-request-available > Fix For: 1.0.0 > > > We should add the record key in MDT when validating the partition stats. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HUDI-7707) Enable bundle validation on Java 8 and 11
[ https://issues.apache.org/jira/browse/HUDI-7707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7707: Sprint: Sprint 2023-04-26 > Enable bundle validation on Java 8 and 11 > - > > Key: HUDI-7707 > URL: https://issues.apache.org/jira/browse/HUDI-7707 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Assignee: Ethan Guo >Priority: Major > Fix For: 0.15.0, 1.0.0 > > Attachments: Screenshot 2024-05-02 at 17.41.02.png > > > Bundle validation with Java 8 and 11 are somehow skipped in GH CI. They > should be enabled. !Screenshot 2024-05-02 at > 17.41.02.png|width=905,height=325! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HUDI-7707) Enable bundle validation on Java 8 and 11
[ https://issues.apache.org/jira/browse/HUDI-7707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7707: Description: Bundle validation with Java 8 and 11 are somehow skipped in GH CI. They should be enabled. !Screenshot 2024-05-02 at 17.41.02.png|width=905,height=325! > Enable bundle validation on Java 8 and 11 > - > > Key: HUDI-7707 > URL: https://issues.apache.org/jira/browse/HUDI-7707 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Assignee: Ethan Guo >Priority: Major > Fix For: 0.15.0, 1.0.0 > > Attachments: Screenshot 2024-05-02 at 17.41.02.png > > > Bundle validation with Java 8 and 11 are somehow skipped in GH CI. They > should be enabled. !Screenshot 2024-05-02 at > 17.41.02.png|width=905,height=325! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HUDI-7707) Enable bundle validation on Java 8 and 11
[ https://issues.apache.org/jira/browse/HUDI-7707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7707: Attachment: Screenshot 2024-05-02 at 17.41.02.png > Enable bundle validation on Java 8 and 11 > - > > Key: HUDI-7707 > URL: https://issues.apache.org/jira/browse/HUDI-7707 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Assignee: Ethan Guo >Priority: Major > Fix For: 0.15.0, 1.0.0 > > Attachments: Screenshot 2024-05-02 at 17.41.02.png > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HUDI-7707) Enable bundle validation on Java 8 and 11
[ https://issues.apache.org/jira/browse/HUDI-7707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo reassigned HUDI-7707: --- Assignee: Ethan Guo > Enable bundle validation on Java 8 and 11 > - > > Key: HUDI-7707 > URL: https://issues.apache.org/jira/browse/HUDI-7707 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Assignee: Ethan Guo >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HUDI-7707) Enable bundle validation on Java 8 and 11
[ https://issues.apache.org/jira/browse/HUDI-7707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7707: Fix Version/s: 0.15.0 1.0.0 > Enable bundle validation on Java 8 and 11 > - > > Key: HUDI-7707 > URL: https://issues.apache.org/jira/browse/HUDI-7707 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Assignee: Ethan Guo >Priority: Major > Fix For: 0.15.0, 1.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HUDI-7707) Enable bundle validation on Java 8 and 11
Ethan Guo created HUDI-7707: --- Summary: Enable bundle validation on Java 8 and 11 Key: HUDI-7707 URL: https://issues.apache.org/jira/browse/HUDI-7707 Project: Apache Hudi Issue Type: Improvement Reporter: Ethan Guo -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [HUDI-7686] Add tests on the util methods for type cast of configuration instances [hudi]
yihua commented on code in PR #11121:
URL: https://github.com/apache/hudi/pull/11121#discussion_r1588552025
##
hudi-io/src/test/java/org/apache/hudi/io/storage/BaseTestStorageConfiguration.java:
##
@@ -71,13 +72,25 @@ public abstract class BaseTestStorageConfiguration {
@Test
public void testConstructorNewInstanceUnwrapCopy() {
-T conf = getConf(EMPTY_MAP);
+T conf = getConf(prepareConfigs());
StorageConfiguration storageConf = getStorageConfiguration(conf);
StorageConfiguration newStorageConf = storageConf.newInstance();
+Class unwrapperConfClass = storageConf.unwrap().getClass();
assertNotSame(storageConf, newStorageConf);
+validateConfigs(newStorageConf);
assertNotSame(storageConf.unwrap(), newStorageConf.unwrap());
assertSame(storageConf.unwrap(), storageConf.unwrap());
+assertSame(storageConf.unwrap(), storageConf.unwrapAs(unwrapperConfClass));
assertNotSame(storageConf.unwrap(), storageConf.unwrapCopy());
+validateConfigs(getStorageConfiguration(storageConf.unwrapCopy()));
+assertNotSame(storageConf.unwrap(),
storageConf.unwrapCopyAs(unwrapperConfClass));
+validateConfigs(getStorageConfiguration((T)
storageConf.unwrapCopyAs(unwrapperConfClass)));
Review Comment:
Addressed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]
hudi-bot commented on PR #11130: URL: https://github.com/apache/hudi/pull/11130#issuecomment-2091943286 ## CI report: * edf2bf30a2ddbd48db9452f34b1ac716bd2ebe18 UNKNOWN * b1598f5861c2b90da91ad33dc360533728ef7163 UNKNOWN * 9a2450a1bb4454ddc2c86791ce112201f431627a UNKNOWN * 9451df2ad814d4d17a38cee04309a35d2c94b6e7 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23631) * 8105ef96648ad16ec61237a974dbed9e6a2d2c8f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23632) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]
hudi-bot commented on PR #11130: URL: https://github.com/apache/hudi/pull/11130#issuecomment-2091937900 ## CI report: * edf2bf30a2ddbd48db9452f34b1ac716bd2ebe18 UNKNOWN * b1598f5861c2b90da91ad33dc360533728ef7163 UNKNOWN * 9a2450a1bb4454ddc2c86791ce112201f431627a UNKNOWN * 9451df2ad814d4d17a38cee04309a35d2c94b6e7 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23631) * 8105ef96648ad16ec61237a974dbed9e6a2d2c8f UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]
yihua commented on code in PR #11130:
URL: https://github.com/apache/hudi/pull/11130#discussion_r1588541644
##
hudi-spark-datasource/hudi-spark3.5.x/src/test/java/org/apache/hudi/spark3/internal/TestReflectUtil.java:
##
@@ -42,7 +44,7 @@ public void testDataSourceWriterExtraCommitMetadata() throws
Exception {
InsertIntoStatement newStatment = ReflectUtil.createInsertInto(
statement.table(),
statement.partitionSpec(),
-scala.collection.immutable.List.empty(),
+((scala.collection.immutable.Seq)
scala.collection.immutable.Seq$.MODULE$.empty()).toSeq(),
Review Comment:
FYI this is done in #11135 .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (HUDI-7695) Add docs on Spark 3.5 and Scala 2.13
[ https://issues.apache.org/jira/browse/HUDI-7695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7695: Sprint: Sprint 2023-04-26 > Add docs on Spark 3.5 and Scala 2.13 > > > Key: HUDI-7695 > URL: https://issues.apache.org/jira/browse/HUDI-7695 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Assignee: Ethan Guo >Priority: Major > Fix For: 0.15.0, 1.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HUDI-7702) Remove unused method in ReflectUtil
[ https://issues.apache.org/jira/browse/HUDI-7702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7702: Sprint: Sprint 2023-04-26 > Remove unused method in ReflectUtil > --- > > Key: HUDI-7702 > URL: https://issues.apache.org/jira/browse/HUDI-7702 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Assignee: Ethan Guo >Priority: Major > Labels: pull-request-available > Fix For: 0.15.0, 1.0.0 > > > ReflectUtil#createInsertInto is no longer used in the repo and causes issue > for Scala 2.13 support. We should remove the unused method. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HUDI-7705) Column name is wrong when generating partition stats index key
[ https://issues.apache.org/jira/browse/HUDI-7705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7705: Sprint: Sprint 2023-04-26 > Column name is wrong when generating partition stats index key > -- > > Key: HUDI-7705 > URL: https://issues.apache.org/jira/browse/HUDI-7705 > Project: Apache Hudi > Issue Type: Bug >Reporter: Ethan Guo >Assignee: Sagar Sumit >Priority: Major > Fix For: 1.0.0 > > Attachments: Screenshot 2024-05-02 at 11.10.59.png > > > When running the test "Test hudi_metadata Table-Valued Function For > PARTITION_STATS index" in TestHoodieTableValuedFunction, the column name is > wrong when generating partition stats index key (see the screenshot below). > The "price" column is used instead of the partition column "ts". The causes > the record key of partition stats index record to have wrong prefix due to > different column name used. > !Screenshot 2024-05-02 at 11.10.59.png|width=933,height=421! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HUDI-7706) Improve validation in PARTITION_STATS index test
[ https://issues.apache.org/jira/browse/HUDI-7706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7706: Sprint: Sprint 2023-04-26 > Improve validation in PARTITION_STATS index test > > > Key: HUDI-7706 > URL: https://issues.apache.org/jira/browse/HUDI-7706 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Assignee: Ethan Guo >Priority: Major > Labels: pull-request-available > Fix For: 1.0.0 > > > We should add the record key in MDT when validating the partition stats. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HUDI-7706) Improve validation in PARTITION_STATS index test
[ https://issues.apache.org/jira/browse/HUDI-7706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7706: Story Points: 1 > Improve validation in PARTITION_STATS index test > > > Key: HUDI-7706 > URL: https://issues.apache.org/jira/browse/HUDI-7706 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Assignee: Ethan Guo >Priority: Major > Labels: pull-request-available > Fix For: 1.0.0 > > > We should add the record key in MDT when validating the partition stats. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]
hudi-bot commented on PR #11130: URL: https://github.com/apache/hudi/pull/11130#issuecomment-2091896712 ## CI report: * edf2bf30a2ddbd48db9452f34b1ac716bd2ebe18 UNKNOWN * b1598f5861c2b90da91ad33dc360533728ef7163 UNKNOWN * 9a2450a1bb4454ddc2c86791ce112201f431627a UNKNOWN * 9451df2ad814d4d17a38cee04309a35d2c94b6e7 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23631) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7587] Make bundle dependencies for storage abstraction in correct order [hudi]
hudi-bot commented on PR #11131: URL: https://github.com/apache/hudi/pull/11131#issuecomment-2091884462 ## CI report: * 834aad2a8b073a221e68fb3c960200f684b84dfd Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23630) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]
hudi-bot commented on PR #11130: URL: https://github.com/apache/hudi/pull/11130#issuecomment-2091837042 ## CI report: * edf2bf30a2ddbd48db9452f34b1ac716bd2ebe18 UNKNOWN * b1598f5861c2b90da91ad33dc360533728ef7163 UNKNOWN * 9a2450a1bb4454ddc2c86791ce112201f431627a UNKNOWN * 35803650a3fd3ff6f5cfa4a372a592a18d04bdcc Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23628) * 9451df2ad814d4d17a38cee04309a35d2c94b6e7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23631) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]
hudi-bot commented on PR #11130: URL: https://github.com/apache/hudi/pull/11130#issuecomment-2091829624 ## CI report: * edf2bf30a2ddbd48db9452f34b1ac716bd2ebe18 UNKNOWN * b1598f5861c2b90da91ad33dc360533728ef7163 UNKNOWN * 9a2450a1bb4454ddc2c86791ce112201f431627a UNKNOWN * 35803650a3fd3ff6f5cfa4a372a592a18d04bdcc Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23628) * 9451df2ad814d4d17a38cee04309a35d2c94b6e7 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7587] Make bundle dependencies for storage abstraction in correct order [hudi]
hudi-bot commented on PR #11131: URL: https://github.com/apache/hudi/pull/11131#issuecomment-2091820661 ## CI report: * c0a81f2890f9b066738fdf74cad9edf79cae0fda Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23625) * 834aad2a8b073a221e68fb3c960200f684b84dfd Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23630) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [SUPPORT] is it possible to read/write hudi files with another programming language? [hudi]
xushiyan commented on issue #7446: URL: https://github.com/apache/hudi/issues/7446#issuecomment-2091776935 @vinothchandar yes. gonna take care of repo logistics and dev setup to make the repo ready for new contributors. Also preparing issues to work on. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7587] Make bundle dependencies for storage abstraction in correct order [hudi]
hudi-bot commented on PR #11131: URL: https://github.com/apache/hudi/pull/11131#issuecomment-2091675567 ## CI report: * c0a81f2890f9b066738fdf74cad9edf79cae0fda Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23625) * 834aad2a8b073a221e68fb3c960200f684b84dfd UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]
hudi-bot commented on PR #11130: URL: https://github.com/apache/hudi/pull/11130#issuecomment-2091632448 ## CI report: * edf2bf30a2ddbd48db9452f34b1ac716bd2ebe18 UNKNOWN * b1598f5861c2b90da91ad33dc360533728ef7163 UNKNOWN * bf6aaf244d52cc66e7c93d7a8a02502e9941 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23626) * 9a2450a1bb4454ddc2c86791ce112201f431627a UNKNOWN * 35803650a3fd3ff6f5cfa4a372a592a18d04bdcc Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23628) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]
hudi-bot commented on PR #11130: URL: https://github.com/apache/hudi/pull/11130#issuecomment-2091590841 ## CI report: * edf2bf30a2ddbd48db9452f34b1ac716bd2ebe18 UNKNOWN * b1598f5861c2b90da91ad33dc360533728ef7163 UNKNOWN * bf6aaf244d52cc66e7c93d7a8a02502e9941 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23626) * 9a2450a1bb4454ddc2c86791ce112201f431627a UNKNOWN * 35803650a3fd3ff6f5cfa4a372a592a18d04bdcc UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7706] Improve validation in PARTITION_STATS index test [hudi]
yihua merged PR #11141: URL: https://github.com/apache/hudi/pull/11141 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(hudi) branch master updated (156e7604f8d -> 65f4b594c28)
This is an automated email from the ASF dual-hosted git repository. yihua pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from 156e7604f8d [HUDI-4372] Enable matadata table by default for flink (#11124) add 65f4b594c28 [HUDI-7706] Improve validation in PARTITION_STATS index test (#11141) No new revisions were added by this update. Summary of changes: .../hudi/dml/TestHoodieTableValuedFunction.scala | 27 ++ 1 file changed, 18 insertions(+), 9 deletions(-)
Re: [PR] [HUDI-7587] Make bundle dependencies for storage abstraction in correct order [hudi]
hudi-bot commented on PR #11131: URL: https://github.com/apache/hudi/pull/11131#issuecomment-2091556390 ## CI report: * c0a81f2890f9b066738fdf74cad9edf79cae0fda Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23625) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7706] Improve validation in PARTITION_STATS index test [hudi]
hudi-bot commented on PR #11141: URL: https://github.com/apache/hudi/pull/11141#issuecomment-2091556714 ## CI report: * 3b9f0a272b58a7eb8f63ad20edd047b4aa740ccf Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23624) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]
hudi-bot commented on PR #11130: URL: https://github.com/apache/hudi/pull/11130#issuecomment-2091556140 ## CI report: * edf2bf30a2ddbd48db9452f34b1ac716bd2ebe18 UNKNOWN * b1598f5861c2b90da91ad33dc360533728ef7163 UNKNOWN * bf6aaf244d52cc66e7c93d7a8a02502e9941 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23626) * 9a2450a1bb4454ddc2c86791ce112201f431627a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]
hudi-bot commented on PR #11130: URL: https://github.com/apache/hudi/pull/11130#issuecomment-2091473097 ## CI report: * edf2bf30a2ddbd48db9452f34b1ac716bd2ebe18 UNKNOWN * b1598f5861c2b90da91ad33dc360533728ef7163 UNKNOWN * bf6aaf244d52cc66e7c93d7a8a02502e9941 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23626) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]
hudi-bot commented on PR #11130: URL: https://github.com/apache/hudi/pull/11130#issuecomment-2091461817 ## CI report: * edf2bf30a2ddbd48db9452f34b1ac716bd2ebe18 UNKNOWN * b1598f5861c2b90da91ad33dc360533728ef7163 UNKNOWN * d33ea2a54ba42ccb221156b9013889b7b6b0af94 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23623) * bf6aaf244d52cc66e7c93d7a8a02502e9941 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7587] Make bundle dependencies for storage abstraction in correct order [hudi]
hudi-bot commented on PR #11131: URL: https://github.com/apache/hudi/pull/11131#issuecomment-2091448728 ## CI report: * b88fa88e1a946edf8da8f0686345fe06fd0f55ce Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23621) * c0a81f2890f9b066738fdf74cad9edf79cae0fda Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23625) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]
hudi-bot commented on PR #11130: URL: https://github.com/apache/hudi/pull/11130#issuecomment-2091448670 ## CI report: * edf2bf30a2ddbd48db9452f34b1ac716bd2ebe18 UNKNOWN * b1598f5861c2b90da91ad33dc360533728ef7163 UNKNOWN * d33ea2a54ba42ccb221156b9013889b7b6b0af94 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23623) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7706] Improve validation in PARTITION_STATS index test [hudi]
hudi-bot commented on PR #11141: URL: https://github.com/apache/hudi/pull/11141#issuecomment-2091343302 ## CI report: * 3b9f0a272b58a7eb8f63ad20edd047b4aa740ccf Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23624) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7587] Make bundle dependencies for storage abstraction in correct order [hudi]
hudi-bot commented on PR #11131: URL: https://github.com/apache/hudi/pull/11131#issuecomment-2091343002 ## CI report: * b88fa88e1a946edf8da8f0686345fe06fd0f55ce Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23621) * c0a81f2890f9b066738fdf74cad9edf79cae0fda UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]
hudi-bot commented on PR #11130: URL: https://github.com/apache/hudi/pull/11130#issuecomment-2091342715 ## CI report: * edf2bf30a2ddbd48db9452f34b1ac716bd2ebe18 UNKNOWN * b1598f5861c2b90da91ad33dc360533728ef7163 UNKNOWN * e869465714018ad7085a175529dfc8f700ee867c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23605) * 6c9451c549a524dec16538f30fc7942517b186e9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23620) * 2e33f2c6c1606a7e602ecd60455ccdbc80a1bb94 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23622) * d33ea2a54ba42ccb221156b9013889b7b6b0af94 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23623) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7706] Improve validation in PARTITION_STATS index test [hudi]
hudi-bot commented on PR #11141: URL: https://github.com/apache/hudi/pull/11141#issuecomment-2091302414 ## CI report: * 3b9f0a272b58a7eb8f63ad20edd047b4aa740ccf UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]
hudi-bot commented on PR #11130: URL: https://github.com/apache/hudi/pull/11130#issuecomment-2091302055 ## CI report: * edf2bf30a2ddbd48db9452f34b1ac716bd2ebe18 UNKNOWN * b1598f5861c2b90da91ad33dc360533728ef7163 UNKNOWN * e869465714018ad7085a175529dfc8f700ee867c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23605) * 6c9451c549a524dec16538f30fc7942517b186e9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23620) * 2e33f2c6c1606a7e602ecd60455ccdbc80a1bb94 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23622) * d33ea2a54ba42ccb221156b9013889b7b6b0af94 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7587] Make bundle dependencies for storage abstraction in correct order [hudi]
jonvex commented on code in PR #11131:
URL: https://github.com/apache/hudi/pull/11131#discussion_r1588149891
##
hudi-common/src/test/java/org/apache/hudi/common/testutils/HoodieTestUtils.java:
##
@@ -68,7 +68,8 @@ public class HoodieTestUtils {
public static final String[] DEFAULT_PARTITION_PATHS = {"2016/03/15",
"2015/03/16", "2015/03/17"};
public static StorageConfiguration getDefaultStorageConf() {
-return HadoopFSUtils.getStorageConf(new Configuration(false));
+return (StorageConfiguration)
ReflectionUtils.loadClass("org.apache.hudi.storage.hadoop.HadoopStorageConfiguration",
Review Comment:
should make this use hoodiestorageutils
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [MINOR] remove unnecessary lines from java test [hudi]
hudi-bot commented on PR #11139: URL: https://github.com/apache/hudi/pull/11139#issuecomment-2091284961 ## CI report: * 069377621b3112a0280529fb15845afa9d58f991 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23619) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7587] Make bundle dependencies for storage abstraction in correct order [hudi]
hudi-bot commented on PR #11131: URL: https://github.com/apache/hudi/pull/11131#issuecomment-2091284841 ## CI report: * b88fa88e1a946edf8da8f0686345fe06fd0f55ce Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23621) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-6296] Add Scala 2.13 support for Spark 3.5 integration [hudi]
hudi-bot commented on PR #11130: URL: https://github.com/apache/hudi/pull/11130#issuecomment-2091284743 ## CI report: * edf2bf30a2ddbd48db9452f34b1ac716bd2ebe18 UNKNOWN * b1598f5861c2b90da91ad33dc360533728ef7163 UNKNOWN * e869465714018ad7085a175529dfc8f700ee867c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23605) * 6c9451c549a524dec16538f30fc7942517b186e9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=23620) * 2e33f2c6c1606a7e602ecd60455ccdbc80a1bb94 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
