(hudi) annotated tag release-1.0.0-beta1-rc1 updated (2bc5518c138 -> e97aa6ccb56)
This is an automated email from the ASF dual-hosted git repository. yihua pushed a change to annotated tag release-1.0.0-beta1-rc1 in repository https://gitbox.apache.org/repos/asf/hudi.git *** WARNING: tag release-1.0.0-beta1-rc1 was modified! *** from 2bc5518c138 (commit) to e97aa6ccb56 (tag) tagging 2bc5518c138d55581e1fd3f7327fc049cab3e77c (commit) replaces hoodie-0.4.7 by Y Ethan Guo on Fri Nov 10 23:59:08 2023 -0800 - Log - release-1.0.0-beta1 -BEGIN PGP SIGNATURE- iQIzBAABCAAdFiEEDE0xZCfsqnGiCtlma+HUVMkPXqUFAmVPNEwACgkQa+HUVMkP XqVzHg//X4bjru5AdLmrhzmvXqer6LPi8kWBp+gqcmGQtAhYlciw92G838jG8xTn 7mndWF9rU612id869frTrEOt3CIAgdQ7m4Jn3PVt2l8QSBChVcJ5LgGKFVHiRQdc YdIKNkrKu3vgw9ip6O2tAUrXCNLPyao2yNb2Z5niWlwa3sIxb3APXv9gE5IO6vGD tT7zaPK0ZI+l1mGSpajwaETe1rSKBYLV98MtX5vr6FYdS7jASdtKxavyfMeCWpwc hw3fE0vpAgB3hkXkyg39KOapsQNjqFVnWKxYf3tpgxpanIhPPMisoS6PZIAq8A36 7/KijkpL0TD9S0NJPz76HhomITceF9QAzDM/FtTwioMVq8bj+BM/E4OccAfWechh fpAxQPVCURljNztgBlcj9kXmyV76nrK2xxajUoDNqaPJKsqtNJ8X6Rez8KHR1Oi4 oZceFKBp4wYucxrTrj5WSVSW3WyLq6EB7RBJ/v3g17X0GF16idENsGOo3h9mkWP9 F+QKKAq5W0hXJgpJ9zCaQhEMONwFCMmJ2KQJqXIKJWjHAEvB0cJxEdiKNYYarrz8 1tjezuMEUCv1X/ciKWHgcXvYNMgP3TCXrAwnzRHFK998aJF4OMLuMvIaS5D4pKNp 4zwrGeCLXrU3j+BxAL+EKJDKGoNgn91Cb5/V1o4VEyka6QiPXhM= =XPzw -END PGP SIGNATURE- --- No new revisions were added by this update. Summary of changes:
[jira] [Closed] (HUDI-7074) Flink incremental query for non-blocking concurrency control
[ https://issues.apache.org/jira/browse/HUDI-7074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit closed HUDI-7074. - Resolution: Fixed > Flink incremental query for non-blocking concurrency control > > > Key: HUDI-7074 > URL: https://issues.apache.org/jira/browse/HUDI-7074 > Project: Apache Hudi > Issue Type: Improvement > Components: flink >Reporter: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 1.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HUDI-6787) Integrate FileGroupReader with HoodieMergeOnReadSnapshotReader and RealtimeCompactedRecordReader for Hive
[ https://issues.apache.org/jira/browse/HUDI-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-6787: -- Fix Version/s: 1.0.0 (was: 1.0.0-beta1) > Integrate FileGroupReader with HoodieMergeOnReadSnapshotReader and > RealtimeCompactedRecordReader for Hive > - > > Key: HUDI-6787 > URL: https://issues.apache.org/jira/browse/HUDI-6787 > Project: Apache Hudi > Issue Type: New Feature >Reporter: Ethan Guo >Priority: Blocker > Fix For: 1.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [HUDI-7074] Flink incremental query for non-blocking concurrency control [hudi]
danny0405 commented on code in PR #10053:
URL: https://github.com/apache/hudi/pull/10053#discussion_r1390137807
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergedReadHandle.java:
##
@@ -99,7 +99,7 @@ private Option getLatestFileSlice() {
&&
hoodieTable.getMetaClient().getCommitsTimeline().filterCompletedInstants().lastInstant().isPresent())
{
return Option.fromJavaOptional(hoodieTable
.getHoodieView()
-
.getLatestMergedFileSlicesBeforeOrOn(partitionPathFileIDPair.getLeft(),
instantTime)
+ .getLatestFileSlices(partitionPathFileIDPair.getLeft())
Review Comment:
It won't, the method intends to return the latest commit but it invokes the
`getLatestMergedFileSlicesBeforeOrOn` which has some problems on NB-CC
scenarios.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
(hudi) branch master updated (c260bce03ec -> 796ac8697e7)
This is an automated email from the ASF dual-hosted git repository. danny0405 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from c260bce03ec [HUDI-7076] Turn on new features by default through configs for 1.0.0-beta1 (#9998) add 796ac8697e7 [HUDI-7074] Flink incremental query for non-blocking concurrency control (#10053) No new revisions were added by this update. Summary of changes: .../org/apache/hudi/io/HoodieMergedReadHandle.java | 2 +- .../hudi/client/TestJavaHoodieBackedMetadata.java | 21 - .../functional/TestHoodieBackedMetadata.java | 26 +-- .../hudi/metadata/HoodieTableMetadataUtil.java | 4 +- .../hudi/sink/bootstrap/BootstrapOperator.java | 2 +- .../partitioner/profile/DeltaWriteProfile.java | 2 +- .../apache/hudi/source/IncrementalInputSplits.java | 32 +++--- .../apache/hudi/table/format/TestInputFormat.java | 50 ++ .../test/java/org/apache/hudi/utils/TestUtils.java | 11 + 9 files changed, 135 insertions(+), 15 deletions(-)
Re: [PR] [HUDI-7074] Flink incremental query for non-blocking concurrency control [hudi]
danny0405 merged PR #10053: URL: https://github.com/apache/hudi/pull/10053 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7074] Flink incremental query for non-blocking concurrency control [hudi]
danny0405 commented on code in PR #10053:
URL: https://github.com/apache/hudi/pull/10053#discussion_r1390137560
##
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##
@@ -1000,7 +1000,9 @@ private static List
getPartitionFileSlices(HoodieTableMetaClient meta
if (mergeFileSlices) {
if
(metaClient.getActiveTimeline().filterCompletedInstants().lastInstant().isPresent())
{
fileSliceStream = fsView.getLatestMergedFileSlicesBeforeOrOn(
-partition,
metaClient.getActiveTimeline().filterCompletedInstants().lastInstant().get().getTimestamp());
+// including pending compaction instant as the last instant so
that the finished delta commits
+// that start earlier than the compaction can be queried.
+partition,
metaClient.getActiveTimeline().filterCompletedAndCompactionInstants().lastInstant().get().getTimestamp());
Review Comment:
yeah, just additionally including the pending compaction instants because
now we do file slicing based on completion time, we should use compaction
instant to filter out the file slices if there are del_commit with smaller
start time but greater completion time (than the compaction)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7074] Flink incremental query for non-blocking concurrency control [hudi]
codope commented on code in PR #10053:
URL: https://github.com/apache/hudi/pull/10053#discussion_r1390135823
##
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##
@@ -1000,7 +1000,9 @@ private static List
getPartitionFileSlices(HoodieTableMetaClient meta
if (mergeFileSlices) {
if
(metaClient.getActiveTimeline().filterCompletedInstants().lastInstant().isPresent())
{
fileSliceStream = fsView.getLatestMergedFileSlicesBeforeOrOn(
-partition,
metaClient.getActiveTimeline().filterCompletedInstants().lastInstant().get().getTimestamp());
+// including pending compaction instant as the last instant so
that the finished delta commits
+// that start earlier than the compaction can be queried.
+partition,
metaClient.getActiveTimeline().filterCompletedAndCompactionInstants().lastInstant().get().getTimestamp());
Review Comment:
you mean to filter **completed** compaction instants and not the **pending**
right? Comment above says pending compaction.
##
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergedReadHandle.java:
##
@@ -99,7 +99,7 @@ private Option getLatestFileSlice() {
&&
hoodieTable.getMetaClient().getCommitsTimeline().filterCompletedInstants().lastInstant().isPresent())
{
return Option.fromJavaOptional(hoodieTable
.getHoodieView()
-
.getLatestMergedFileSlicesBeforeOrOn(partitionPathFileIDPair.getLeft(),
instantTime)
+ .getLatestFileSlices(partitionPathFileIDPair.getLeft())
Review Comment:
can it return uncommitted file slices? I think it won't but just to confirm.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-7082] Add Flink 1.14 and Spark 3.13 docker image script [hudi]
hudi-bot commented on PR #10066: URL: https://github.com/apache/hudi/pull/10066#issuecomment-1806697318 ## CI report: * 8c32f7bc02016c301288575fabc9db2eac46aaab Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20850) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-5936] Fix serialization problem when FileStatus is not serializable [hudi]
hudi-bot commented on PR #10065: URL: https://github.com/apache/hudi/pull/10065#issuecomment-1806674537 ## CI report: * dc9735093064e2f15277330d56012cf7835604da UNKNOWN * f7d0cf8adf8e80204fe9de5f231d062ae43e6787 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20848) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
svn commit: r65210 - in /dev/hudi/hudi-1.0.0-beta1-rc1: ./ hudi-1.0.0-beta1-rc1.src.tgz hudi-1.0.0-beta1-rc1.src.tgz.asc hudi-1.0.0-beta1-rc1.src.tgz.sha512
Author: yihua Date: Sat Nov 11 04:24:11 2023 New Revision: 65210 Log: Add Apache Hudi 1.0.0-beta1 RC1 source release Added: dev/hudi/hudi-1.0.0-beta1-rc1/ dev/hudi/hudi-1.0.0-beta1-rc1/hudi-1.0.0-beta1-rc1.src.tgz (with props) dev/hudi/hudi-1.0.0-beta1-rc1/hudi-1.0.0-beta1-rc1.src.tgz.asc dev/hudi/hudi-1.0.0-beta1-rc1/hudi-1.0.0-beta1-rc1.src.tgz.sha512 Added: dev/hudi/hudi-1.0.0-beta1-rc1/hudi-1.0.0-beta1-rc1.src.tgz == Binary file - no diff available. Propchange: dev/hudi/hudi-1.0.0-beta1-rc1/hudi-1.0.0-beta1-rc1.src.tgz -- svn:mime-type = application/octet-stream Added: dev/hudi/hudi-1.0.0-beta1-rc1/hudi-1.0.0-beta1-rc1.src.tgz.asc == --- dev/hudi/hudi-1.0.0-beta1-rc1/hudi-1.0.0-beta1-rc1.src.tgz.asc (added) +++ dev/hudi/hudi-1.0.0-beta1-rc1/hudi-1.0.0-beta1-rc1.src.tgz.asc Sat Nov 11 04:24:11 2023 @@ -0,0 +1,16 @@ +-BEGIN PGP SIGNATURE- + +iQIzBAABCAAdFiEEiIqTQeYA64VQqs1e+xt1BPf3cMkFAmVPAB4ACgkQ+xt1BPf3 +cMmutg/+J4Ih8KoHPpAJp4CwXRjgSlRRK8mHYxA9w2xXW9qo8baD4Sw/vU+jG9// +Howc+RD4NPasN+jBaxvYMAqIJoGFzHQY3mcu+axMF6mVSfS4JsBZgET1Tzqh9SSF +fSxOHJQUxbhtl/NQkveWmNlIapUpFxgkgerXOvRibyrb29WZGl55ZAGQVG1Jl3zR +YIRe2Rvtqxi4I7ahkm/zDiwJn11ldEVoTeKLQ0+n/95IICdRwpTiYyhcvU/YlF+9 +OoT7zAmhJmGC7aFcdh5veJr0w1aeFLYqyxmh3kEukAIrgQRRqnCBYLk0whsiG0r1 +NEZ7/RfnlV8T5yc8x8AcaxzZo+Qdv+iOcHc8UeNm7Eg14GOrkfFEQZPRkLuyZgUn +yZmRh44KCowVLVGSPujjATLNmv5XT+7TimwhO7/DSMZ9Io652Uzz1uQ3H7bQ8GAk +RHvFyjdHarX+evCHK5YbTzQ/0rAJuEcUS9xjJyWZKrb9CnyDxIOzDOVixOYRrxnv +RJ/Dti3Qhf9LYeHvSKzP2pDWbDoRaC4va3JfuxI6LqOZVrAl3vBraCjyKXeZLIVU +W3D7ygklpSa/yZ6X90SfqxNuJZLbJkA3gHsVBzNmKFtH6Rw1NOfc2x3PZiIMPnXw +xuVhoM41nJwn2a5PQ1Qq3OSLihB8YCKKILFV/Ncv/IBpLkmrI7s= +=BJNM +-END PGP SIGNATURE- Added: dev/hudi/hudi-1.0.0-beta1-rc1/hudi-1.0.0-beta1-rc1.src.tgz.sha512 == --- dev/hudi/hudi-1.0.0-beta1-rc1/hudi-1.0.0-beta1-rc1.src.tgz.sha512 (added) +++ dev/hudi/hudi-1.0.0-beta1-rc1/hudi-1.0.0-beta1-rc1.src.tgz.sha512 Sat Nov 11 04:24:11 2023 @@ -0,0 +1 @@ +80f1894c5199b1ea251b5890b52cf66258f0a5398d0cf5a0773dcfd2b592f12d2e7fb8968675d5a936effc727335aee9f943c6c5a8f38413cc6718f11af28c18 hudi-1.0.0-beta1-rc1.src.tgz
(hudi) branch master updated: [HUDI-7076] Turn on new features by default through configs for 1.0.0-beta1 (#9998)
This is an automated email from the ASF dual-hosted git repository.
yihua pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/master by this push:
new c260bce03ec [HUDI-7076] Turn on new features by default through
configs for 1.0.0-beta1 (#9998)
c260bce03ec is described below
commit c260bce03ec0b1abbd14af1e3ef9617bbae9e80a
Author: Sagar Sumit
AuthorDate: Sat Nov 11 09:12:47 2023 +0530
[HUDI-7076] Turn on new features by default through configs for 1.0.0-beta1
(#9998)
This commit enables the following new features by default through configs:
- Write record positions to MOR log data blocks
(`hoodie.write.record.positions`)
- Enable partial updates when possible for Spark SQL MERGE INTO statement
(`hoodie.spark.sql.merge.into.partial.updates`)
- Use new file group reader for MOR snapshot queries
(`hoodie.file.group.reader.enabled`)
- Use new Hudi Spark parquet file format for various types of queries
(`hoodie.datasource.read.use.new.parquet.file.format`)
-
Co-authored-by: Y Ethan Guo
---
.../org/apache/hudi/config/HoodieWriteConfig.java | 2 +-
.../hudi/common/config/HoodieReaderConfig.java | 2 +-
.../scala/org/apache/hudi/DataSourceOptions.scala | 4 +-
.../functional/TestParquetColumnProjection.scala | 51 --
.../hudi/functional/TestSparkDataSource.scala | 3 ++
.../apache/spark/sql/hudi/TestInsertTable.scala| 4 +-
.../hudi/TestNestedSchemaPruningOptimization.scala | 6 +--
.../TestHoodiePruneFileSourcePartitions.scala | 6 +--
.../utilities/sources/TestHoodieIncrSource.java| 2 +
9 files changed, 44 insertions(+), 36 deletions(-)
diff --git
a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java
b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java
index cd8f9f6b629..6a36e5025bc 100644
---
a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java
+++
b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java
@@ -751,7 +751,7 @@ public class HoodieWriteConfig extends HoodieConfig {
public static final ConfigProperty WRITE_RECORD_POSITIONS =
ConfigProperty
.key("hoodie.write.record.positions")
- .defaultValue(false)
+ .defaultValue(true)
.markAdvanced()
.sinceVersion("1.0.0")
.withDocumentation("Whether to write record positions to the block
header for data blocks containing updates and delete blocks. "
diff --git
a/hudi-common/src/main/java/org/apache/hudi/common/config/HoodieReaderConfig.java
b/hudi-common/src/main/java/org/apache/hudi/common/config/HoodieReaderConfig.java
index c572cc21adc..20e745d7a9a 100644
---
a/hudi-common/src/main/java/org/apache/hudi/common/config/HoodieReaderConfig.java
+++
b/hudi-common/src/main/java/org/apache/hudi/common/config/HoodieReaderConfig.java
@@ -54,7 +54,7 @@ public class HoodieReaderConfig extends HoodieConfig {
public static final ConfigProperty FILE_GROUP_READER_ENABLED =
ConfigProperty
.key("hoodie.file.group.reader.enabled")
- .defaultValue(false)
+ .defaultValue(true)
.markAdvanced()
.sinceVersion("1.0.0")
.withDocumentation("Use engine agnostic file group reader if enabled");
diff --git
a/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala
b/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala
index dc54825ac90..efa9c9e692f 100644
---
a/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala
+++
b/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala
@@ -88,7 +88,7 @@ object DataSourceReadOptions {
val USE_NEW_HUDI_PARQUET_FILE_FORMAT: ConfigProperty[String] = ConfigProperty
.key("hoodie.datasource.read.use.new.parquet.file.format")
-.defaultValue("false")
+.defaultValue("true")
.markAdvanced()
.sinceVersion("0.14.0")
.withDocumentation("Read using the new Hudi parquet file format. The new
Hudi parquet file format is " +
@@ -558,7 +558,7 @@ object DataSourceWriteOptions {
val ENABLE_MERGE_INTO_PARTIAL_UPDATES: ConfigProperty[Boolean] =
ConfigProperty
.key("hoodie.spark.sql.merge.into.partial.updates")
-.defaultValue(false)
+.defaultValue(true)
.markAdvanced()
.sinceVersion("1.0.0")
.withDocumentation("Whether to write partial updates to the data blocks
containing updates "
diff --git
a/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestParquetColumnProjection.scala
b/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestParquetColumnProjection.scala
index ee1edbcccb2..6ff7e5681e6 100644
---
Re: [PR] [HUDI-7076] Turn on new features by default through configs for 1.0.0-beta1 [hudi]
yihua merged PR #9998: URL: https://github.com/apache/hudi/pull/9998 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7076] Turn on new features by default through configs for 1.0.0-beta1 [hudi]
yihua commented on PR #9998: URL: https://github.com/apache/hudi/pull/9998#issuecomment-1806641779 The fourth task (`UT FT other modules`) in Azure CI passes in #10055 based on the same set of changes. https://github.com/apache/hudi/assets/2497195/74e488c8-aaa3-45b9-9f3c-9a3227dc81c9;> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7082] Add Flink 1.14 and Spark 3.13 docker image script [hudi]
hudi-bot commented on PR #10066: URL: https://github.com/apache/hudi/pull/10066#issuecomment-1806641536 ## CI report: * 8c32f7bc02016c301288575fabc9db2eac46aaab Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20850) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR] Fix testRenamePartition [hudi]
hudi-bot commented on PR #10064: URL: https://github.com/apache/hudi/pull/10064#issuecomment-1806641517 ## CI report: * 4ab080293d7d6c5d93e0353397fe516ec47dd9a5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20843) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-5936] Fix serialization problem when FileStatus is not serializable [hudi]
hudi-bot commented on PR #10065: URL: https://github.com/apache/hudi/pull/10065#issuecomment-1806616830 ## CI report: * 2c42b4127fa90eb1f5de9948b5a6bec9ed27662e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20846) * dc9735093064e2f15277330d56012cf7835604da UNKNOWN * f7d0cf8adf8e80204fe9de5f231d062ae43e6787 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20848) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7082] Add Flink 1.14 and Spark 3.13 docker image script [hudi]
hudi-bot commented on PR #10066: URL: https://github.com/apache/hudi/pull/10066#issuecomment-1806638295 ## CI report: * 8c32f7bc02016c301288575fabc9db2eac46aaab UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7076] Turn on new features by default through configs for 1.0.0-beta1 [hudi]
hudi-bot commented on PR #9998: URL: https://github.com/apache/hudi/pull/9998#issuecomment-1806638210 ## CI report: * d1f415b1400e0ff853e38b48fdaa7a9fe7aa44d3 UNKNOWN * f7f4a9e3342020e4a1472f5a3b59d25f9c1ec090 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20842) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR][TESTING] Enable configs testing 3 [hudi]
hudi-bot commented on PR #10055: URL: https://github.com/apache/hudi/pull/10055#issuecomment-1806635356 ## CI report: * bfebcd456acf943c525ca2830ab864098a802e9a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20845) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7072] Remove support for Flink 1.13 [hudi]
danny0405 commented on PR #10052: URL: https://github.com/apache/hudi/pull/10052#issuecomment-1806635003 Can you reference PR: https://github.com/apache/hudi/pull/5072, the redundants between 1.13 and 1.14 can be moved back to `hudi-flink` module I think. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [HUDI-7082] Add Flink 1.14 and Spark 3.13 docker image script [hudi]
danny0405 opened a new pull request, #10066: URL: https://github.com/apache/hudi/pull/10066 ### Change Logs We are gonna remove the support for flink 1.13, so update the bundle validation scripts. ### Impact none ### Risk level (write none, low medium or high below) none ### Documentation Update _Describe any necessary documentation update if there is any new feature, config, or user-facing change_ - _The config description must be updated if new configs are added or the default value of the configs are changed_ - _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make changes to the website._ ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Change Logs and Impact were stated clearly - [ ] Adequate tests were added if applicable - [ ] CI passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-7082) Add Flink 1.14 and Spark 3.13 docker image script
[ https://issues.apache.org/jira/browse/HUDI-7082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-7082: - Labels: pull-request-available (was: ) > Add Flink 1.14 and Spark 3.13 docker image script > - > > Key: HUDI-7082 > URL: https://issues.apache.org/jira/browse/HUDI-7082 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Danny Chen >Priority: Major > Labels: pull-request-available > Fix For: 1.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HUDI-7082) Add Flink 1.14 and Spark 3.13 docker image script
Danny Chen created HUDI-7082: Summary: Add Flink 1.14 and Spark 3.13 docker image script Key: HUDI-7082 URL: https://issues.apache.org/jira/browse/HUDI-7082 Project: Apache Hudi Issue Type: Improvement Reporter: Danny Chen Fix For: 1.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [HUDI-5936] Fix serialization problem when FileStatus is not serializable [hudi]
hudi-bot commented on PR #10065: URL: https://github.com/apache/hudi/pull/10065#issuecomment-1806615305 ## CI report: * 2c42b4127fa90eb1f5de9948b5a6bec9ed27662e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20846) * dc9735093064e2f15277330d56012cf7835604da UNKNOWN * f7d0cf8adf8e80204fe9de5f231d062ae43e6787 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-5936] Fix serialization problem when FileStatus is not serializable [hudi]
hudi-bot commented on PR #10065: URL: https://github.com/apache/hudi/pull/10065#issuecomment-1806604213 ## CI report: * 2c42b4127fa90eb1f5de9948b5a6bec9ed27662e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20846) * dc9735093064e2f15277330d56012cf7835604da UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(hudi) branch master updated: [MINOR] Disable reader for test with enum (#10061)
This is an automated email from the ASF dual-hosted git repository.
yihua pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/master by this push:
new 558281ed430 [MINOR] Disable reader for test with enum (#10061)
558281ed430 is described below
commit 558281ed4303756ad7a00331e1568dbb107f8571
Author: Jon Vexler
AuthorDate: Fri Nov 10 19:42:45 2023 -0500
[MINOR] Disable reader for test with enum (#10061)
Co-authored-by: Jonathan Vexler <=>
Co-authored-by: Y Ethan Guo
---
.../hudi/utilities/sources/HoodieIncrSource.java | 20
.../hudi/utilities/sources/TestHoodieIncrSource.java | 3 +++
2 files changed, 23 insertions(+)
diff --git
a/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java
b/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java
index fa316cf806f..aafd4c9e3b5 100644
---
a/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java
+++
b/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java
@@ -19,9 +19,11 @@
package org.apache.hudi.utilities.sources;
import org.apache.hudi.DataSourceReadOptions;
+import org.apache.hudi.common.config.HoodieReaderConfig;
import org.apache.hudi.common.config.TypedProperties;
import org.apache.hudi.common.model.HoodieRecord;
import
org.apache.hudi.common.table.timeline.TimelineUtils.HollowCommitHandling;
+import org.apache.hudi.common.util.CollectionUtils;
import org.apache.hudi.common.util.Option;
import org.apache.hudi.common.util.ReflectionUtils;
import org.apache.hudi.common.util.collection.Pair;
@@ -38,6 +40,9 @@ import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
import static org.apache.hudi.DataSourceReadOptions.BEGIN_INSTANTTIME;
import static org.apache.hudi.DataSourceReadOptions.END_INSTANTTIME;
@@ -58,6 +63,10 @@ import static
org.apache.hudi.utilities.sources.helpers.IncrSourceHelper.getHoll
public class HoodieIncrSource extends RowSource {
private static final Logger LOG =
LoggerFactory.getLogger(HoodieIncrSource.class);
+ public static final Set HOODIE_INCR_SOURCE_READ_OPT_KEYS =
+ CollectionUtils.createImmutableSet(
+ "hoodie.datasource.read.use.new.parquet.file.format",
+ HoodieReaderConfig.FILE_GROUP_READER_ENABLED.key());
private final Option snapshotLoadQuerySplitter;
public static class Config {
@@ -128,10 +137,19 @@ public class HoodieIncrSource extends RowSource {
HoodieIncrSourceConfig.HOODIE_DROP_ALL_META_FIELDS_FROM_SOURCE.defaultValue();
}
+ private final Map readOpts = new HashMap<>();
+
public HoodieIncrSource(TypedProperties props, JavaSparkContext
sparkContext, SparkSession sparkSession,
SchemaProvider schemaProvider) {
super(props, sparkContext, sparkSession, schemaProvider);
+for (Object key : props.keySet()) {
+ String keyString = key.toString();
+ if (HOODIE_INCR_SOURCE_READ_OPT_KEYS.contains(keyString)) {
+readOpts.put(keyString, props.getString(key.toString()));
+ }
+}
+
this.snapshotLoadQuerySplitter =
Option.ofNullable(props.getString(SNAPSHOT_LOAD_QUERY_SPLITTER_CLASS_NAME,
null))
.map(className -> (SnapshotLoadQuerySplitter)
ReflectionUtils.loadClass(className,
new Class[] {TypedProperties.class}, props));
@@ -181,6 +199,7 @@ public class HoodieIncrSource extends RowSource {
// Do Incr pull. Set end instant if available
if (queryInfo.isIncremental()) {
source = sparkSession.read().format("org.apache.hudi")
+ .options(readOpts)
.option(QUERY_TYPE().key(), QUERY_TYPE_INCREMENTAL_OPT_VAL())
.option(BEGIN_INSTANTTIME().key(), queryInfo.getStartInstant())
.option(END_INSTANTTIME().key(), queryInfo.getEndInstant())
@@ -192,6 +211,7 @@ public class HoodieIncrSource extends RowSource {
} else {
// if checkpoint is missing from source table, and if strategy is set to
READ_UPTO_LATEST_COMMIT, we have to issue snapshot query
Dataset snapshot = sparkSession.read().format("org.apache.hudi")
+ .options(readOpts)
.option(DataSourceReadOptions.QUERY_TYPE().key(),
DataSourceReadOptions.QUERY_TYPE_SNAPSHOT_OPT_VAL())
.load(srcPath);
if (snapshotLoadQuerySplitter.isPresent()) {
diff --git
a/hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestHoodieIncrSource.java
b/hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestHoodieIncrSource.java
index d35041592aa..1b534c22c7e 100644
---
a/hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestHoodieIncrSource.java
+++
Re: [PR] [MINOR] Disable reader for test with enum [hudi]
yihua merged PR #10061: URL: https://github.com/apache/hudi/pull/10061 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-5936] Fix serialization problem when FileStatus is not serializable [hudi]
hudi-bot commented on PR #10065: URL: https://github.com/apache/hudi/pull/10065#issuecomment-1806600513 ## CI report: * 2c42b4127fa90eb1f5de9948b5a6bec9ed27662e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20846) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR] Disable reader for test with enum [hudi]
hudi-bot commented on PR #10061: URL: https://github.com/apache/hudi/pull/10061#issuecomment-1806600476 ## CI report: * 1b2a9288f4d10a0c68336eaf97770b165a3bc773 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20839) * f26cf775abaed2874441043af23e2b95d1514855 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-5936] Fix serialization problem when FileStatus is not serializable [hudi]
CTTY commented on PR #10065: URL: https://github.com/apache/hudi/pull/10065#issuecomment-1806598734 Reopened this because this behavior as we are seeing another exception triggered on a different method which is not fixed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-5936] Fix serialization problem when FileStatus is not serializable [hudi]
hudi-bot commented on PR #10065: URL: https://github.com/apache/hudi/pull/10065#issuecomment-1806596299 ## CI report: * 2c42b4127fa90eb1f5de9948b5a6bec9ed27662e UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR][TESTING] Enable configs testing 3 [hudi]
hudi-bot commented on PR #10055: URL: https://github.com/apache/hudi/pull/10055#issuecomment-1806596253 ## CI report: * Unknown: [CANCELED](TBD) * bfebcd456acf943c525ca2830ab864098a802e9a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20845) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7076] Turn on new features by default through configs for 1.0.0-beta1 [hudi]
hudi-bot commented on PR #9998: URL: https://github.com/apache/hudi/pull/9998#issuecomment-1806596160 ## CI report: * d1f415b1400e0ff853e38b48fdaa7a9fe7aa44d3 UNKNOWN * 4cabc36a8ca585b4b1f4ff845145618a70f67174 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20836) * f7f4a9e3342020e4a1472f5a3b59d25f9c1ec090 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20842) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR] disable reader for test with enum [hudi]
yihua commented on code in PR #10061:
URL: https://github.com/apache/hudi/pull/10061#discussion_r1390059295
##
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java:
##
@@ -128,10 +130,16 @@ public static class Config {
HoodieIncrSourceConfig.HOODIE_DROP_ALL_META_FIELDS_FROM_SOURCE.defaultValue();
}
+ private final Map readOpts = new HashMap<>();
+
public HoodieIncrSource(TypedProperties props, JavaSparkContext
sparkContext, SparkSession sparkSession,
SchemaProvider schemaProvider) {
super(props, sparkContext, sparkSession, schemaProvider);
+for (Object key : props.keySet()) {
+ readOpts.put(key.toString(), props.getString(key.toString()));
Review Comment:
Fixed.
##
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestHoodieIncrSource.java:
##
@@ -213,10 +215,9 @@ public void
testHoodieIncrSourceInflightCommitBeforeCompletedCommit(HoodieTableT
}
}
- @ParameterizedTest
- @EnumSource(HoodieTableType.class)
- public void testHoodieIncrSourceWithPendingTableServices(HoodieTableType
tableType) throws IOException {
-this.tableType = tableType;
+ @Test
+ public void testHoodieIncrSourceWithPendingTableServices() throws
IOException {
+this.tableType = MERGE_ON_READ;
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-6658] inject filters for incremental query [hudi]
hudi-bot commented on PR #10063: URL: https://github.com/apache/hudi/pull/10063#issuecomment-1806592047 ## CI report: * dad8fa7cc5f3c3ff65fd0cd9024866083a67db61 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20841) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR][Testing] Test hudi-cli module [hudi]
hudi-bot commented on PR #10060: URL: https://github.com/apache/hudi/pull/10060#issuecomment-1806592026 ## CI report: * dded663cc76f356a4b37941edeaf9dba3adf5c37 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20844) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR][TESTING] Enable configs testing 3 [hudi]
hudi-bot commented on PR #10055: URL: https://github.com/apache/hudi/pull/10055#issuecomment-1806591999 ## CI report: * Unknown: [CANCELED](TBD) * bfebcd456acf943c525ca2830ab864098a802e9a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7076] Turn on new features by default through configs for 1.0.0-beta1 [hudi]
hudi-bot commented on PR #9998: URL: https://github.com/apache/hudi/pull/9998#issuecomment-1806591889 ## CI report: * d1f415b1400e0ff853e38b48fdaa7a9fe7aa44d3 UNKNOWN * 647c783c527a1b18e47feaee3fb7335b37fe5f26 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20834) * 4cabc36a8ca585b4b1f4ff845145618a70f67174 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20836) * f7f4a9e3342020e4a1472f5a3b59d25f9c1ec090 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20842) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR][TESTING] Enable configs testing 3 [hudi]
yihua commented on PR #10055: URL: https://github.com/apache/hudi/pull/10055#issuecomment-1806582292 @hudi-bot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [HUDI-5936] Fix serialization problem when FileStatus is not serializable [hudi]
CTTY opened a new pull request, #10065: URL: https://github.com/apache/hudi/pull/10065 ### Change Logs Original PR: #8190 ### Impact _Describe any public API or user-facing feature change or any performance impact._ ### Risk level (write none, low medium or high below) _If medium or high, explain what verification was done to mitigate the risks._ ### Documentation Update _Describe any necessary documentation update if there is any new feature, config, or user-facing change_ - _The config description must be updated if new configs are added or the default value of the configs are changed_ - _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make changes to the website._ ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Change Logs and Impact were stated clearly - [ ] Adequate tests were added if applicable - [ ] CI passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR] Fix testRenamePartition [hudi]
hudi-bot commented on PR #10064: URL: https://github.com/apache/hudi/pull/10064#issuecomment-1806565749 ## CI report: * 4ab080293d7d6c5d93e0353397fe516ec47dd9a5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20843) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR][Testing] Test hudi-cli module [hudi]
hudi-bot commented on PR #10060: URL: https://github.com/apache/hudi/pull/10060#issuecomment-1806565674 ## CI report: * f70767fb0cf87e383474adc127779f6d31990742 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20838) * dded663cc76f356a4b37941edeaf9dba3adf5c37 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7076] Turn on new features by default through configs for 1.0.0-beta1 [hudi]
hudi-bot commented on PR #9998: URL: https://github.com/apache/hudi/pull/9998#issuecomment-1806565428 ## CI report: * d1f415b1400e0ff853e38b48fdaa7a9fe7aa44d3 UNKNOWN * 647c783c527a1b18e47feaee3fb7335b37fe5f26 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20834) * 4cabc36a8ca585b4b1f4ff845145618a70f67174 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20836) * f7f4a9e3342020e4a1472f5a3b59d25f9c1ec090 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR] Fix testRenamePartition [hudi]
hudi-bot commented on PR #10064: URL: https://github.com/apache/hudi/pull/10064#issuecomment-1806561197 ## CI report: * 4ab080293d7d6c5d93e0353397fe516ec47dd9a5 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR] disable reader for test with enum [hudi]
yihua commented on code in PR #10061:
URL: https://github.com/apache/hudi/pull/10061#discussion_r1390017284
##
hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/HoodieIncrSource.java:
##
@@ -128,10 +130,16 @@ public static class Config {
HoodieIncrSourceConfig.HOODIE_DROP_ALL_META_FIELDS_FROM_SOURCE.defaultValue();
}
+ private final Map readOpts = new HashMap<>();
+
public HoodieIncrSource(TypedProperties props, JavaSparkContext
sparkContext, SparkSession sparkSession,
SchemaProvider schemaProvider) {
super(props, sparkContext, sparkSession, schemaProvider);
+for (Object key : props.keySet()) {
+ readOpts.put(key.toString(), props.getString(key.toString()));
Review Comment:
Instead of putting every properties here, only extract
"FILE_GROUP_READER_ENABLED" and "USE_NEW_HUDI_PARQUET_FILE_FORMAT" here?
##
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestHoodieIncrSource.java:
##
@@ -213,10 +215,9 @@ public void
testHoodieIncrSourceInflightCommitBeforeCompletedCommit(HoodieTableT
}
}
- @ParameterizedTest
- @EnumSource(HoodieTableType.class)
- public void testHoodieIncrSourceWithPendingTableServices(HoodieTableType
tableType) throws IOException {
-this.tableType = tableType;
+ @Test
+ public void testHoodieIncrSourceWithPendingTableServices() throws
IOException {
+this.tableType = MERGE_ON_READ;
Review Comment:
No need to change this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [HUDI-6613] implement inmemory file index to allow for glob paths [hudi]
hudi-bot commented on PR #10062: URL: https://github.com/apache/hudi/pull/10062#issuecomment-1806556765 ## CI report: * 6be57a2edc36106e71c267ad111fa1db4e9bf32a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20840) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR][Testing] Test hudi-cli module [hudi]
hudi-bot commented on PR #10060: URL: https://github.com/apache/hudi/pull/10060#issuecomment-1806556736 ## CI report: * f70767fb0cf87e383474adc127779f6d31990742 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20838) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR][TESTING] Enable configs testing 3 [hudi]
hudi-bot commented on PR #10055: URL: https://github.com/apache/hudi/pull/10055#issuecomment-1806556723 ## CI report: * f70767fb0cf87e383474adc127779f6d31990742 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20837) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20838) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [MINOR] Fix testRenamePartition [hudi]
linliu-code opened a new pull request, #10064: URL: https://github.com/apache/hudi/pull/10064 ### Change Logs Fix unit test: testRenamePartition ### Impact Fix the unit test. ### Risk level (write none, low medium or high below) Low. ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Change Logs and Impact were stated clearly - [ ] Adequate tests were added if applicable - [ ] CI passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-6658] inject filters for incremental query [hudi]
hudi-bot commented on PR #10063: URL: https://github.com/apache/hudi/pull/10063#issuecomment-1806528256 ## CI report: * dad8fa7cc5f3c3ff65fd0cd9024866083a67db61 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20841) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-6613] implement inmemory file index to allow for glob paths [hudi]
hudi-bot commented on PR #10062: URL: https://github.com/apache/hudi/pull/10062#issuecomment-1806528229 ## CI report: * 6be57a2edc36106e71c267ad111fa1db4e9bf32a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20840) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7076] Turn on new features by default through configs for 1.0.0-beta1 [hudi]
hudi-bot commented on PR #9998: URL: https://github.com/apache/hudi/pull/9998#issuecomment-1806527945 ## CI report: * d1f415b1400e0ff853e38b48fdaa7a9fe7aa44d3 UNKNOWN * 647c783c527a1b18e47feaee3fb7335b37fe5f26 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20834) * 4cabc36a8ca585b4b1f4ff845145618a70f67174 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20836) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-6658] inject filters for incremental query [hudi]
hudi-bot commented on PR #10063: URL: https://github.com/apache/hudi/pull/10063#issuecomment-1806523593 ## CI report: * dad8fa7cc5f3c3ff65fd0cd9024866083a67db61 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-6613] implement inmemory file index to allow for glob paths [hudi]
hudi-bot commented on PR #10062: URL: https://github.com/apache/hudi/pull/10062#issuecomment-1806523558 ## CI report: * 6be57a2edc36106e71c267ad111fa1db4e9bf32a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR] disable reader for test with enum [hudi]
hudi-bot commented on PR #10061: URL: https://github.com/apache/hudi/pull/10061#issuecomment-1806518111 ## CI report: * 1b2a9288f4d10a0c68336eaf97770b165a3bc773 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20839) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-6658) Implement MOR Incremental for new file format
[ https://issues.apache.org/jira/browse/HUDI-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-6658: - Labels: pull-request-available (was: ) > Implement MOR Incremental for new file format > - > > Key: HUDI-6658 > URL: https://issues.apache.org/jira/browse/HUDI-6658 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Jonathan Vexler >Assignee: Jonathan Vexler >Priority: Major > Labels: pull-request-available > > Implement MOR Incremental for new file format -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [HUDI-6658] inject filters for incremental query [hudi]
jonvex opened a new pull request, #10063: URL: https://github.com/apache/hudi/pull/10063 ### Change Logs only spark 3.3 currently. want to see how ci runs ### Impact incremental query will work much better including df.count() should be correct. ### Risk level (write none, low medium or high below) low ### Documentation Update N/A ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Change Logs and Impact were stated clearly - [ ] Adequate tests were added if applicable - [ ] CI passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (HUDI-6658) Implement MOR Incremental for new file format
[ https://issues.apache.org/jira/browse/HUDI-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Vexler reassigned HUDI-6658: - Assignee: Jonathan Vexler > Implement MOR Incremental for new file format > - > > Key: HUDI-6658 > URL: https://issues.apache.org/jira/browse/HUDI-6658 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Jonathan Vexler >Assignee: Jonathan Vexler >Priority: Major > > Implement MOR Incremental for new file format -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [HUDI-0000] do not merge. Testing batch reading [hudi]
jonvex closed pull request #10039: [HUDI-] do not merge. Testing batch reading URL: https://github.com/apache/hudi/pull/10039 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [HUDI-6613] implement inmemory file index to allow for glob paths [hudi]
jonvex opened a new pull request, #10062: URL: https://github.com/apache/hudi/pull/10062 ### Change Logs implement inmemory file index to allow for queries with glob paths ### Impact improve new reader ### Risk level (write none, low medium or high below) low ### Documentation Update N/A ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Change Logs and Impact were stated clearly - [ ] Adequate tests were added if applicable - [ ] CI passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-0000] DO NOT MERGE Fix incr errors new reader [hudi]
jonvex closed pull request #9954: [HUDI-] DO NOT MERGE Fix incr errors new reader URL: https://github.com/apache/hudi/pull/9954 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-6613) New file format does not work with in memory index
[ https://issues.apache.org/jira/browse/HUDI-6613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-6613: - Labels: pull-request-available (was: ) > New file format does not work with in memory index > -- > > Key: HUDI-6613 > URL: https://issues.apache.org/jira/browse/HUDI-6613 > Project: Apache Hudi > Issue Type: Improvement > Components: spark >Reporter: Jonathan Vexler >Assignee: Jonathan Vexler >Priority: Major > Labels: pull-request-available > > .option(HoodieIndexConfig.INDEX_TYPE.key, IndexType.INMEMORY.toString) > then the file index is empty so we need to get the files from glob paths -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HUDI-6613) New file format does not work with in memory index
[ https://issues.apache.org/jira/browse/HUDI-6613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Vexler reassigned HUDI-6613: - Assignee: Jonathan Vexler > New file format does not work with in memory index > -- > > Key: HUDI-6613 > URL: https://issues.apache.org/jira/browse/HUDI-6613 > Project: Apache Hudi > Issue Type: Improvement > Components: spark >Reporter: Jonathan Vexler >Assignee: Jonathan Vexler >Priority: Major > > .option(HoodieIndexConfig.INDEX_TYPE.key, IndexType.INMEMORY.toString) > then the file index is empty so we need to get the files from glob paths -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HUDI-6613) New file format does not work with in memory index
[ https://issues.apache.org/jira/browse/HUDI-6613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Vexler updated HUDI-6613: -- Status: In Progress (was: Open) > New file format does not work with in memory index > -- > > Key: HUDI-6613 > URL: https://issues.apache.org/jira/browse/HUDI-6613 > Project: Apache Hudi > Issue Type: Improvement > Components: spark >Reporter: Jonathan Vexler >Assignee: Jonathan Vexler >Priority: Major > > .option(HoodieIndexConfig.INDEX_TYPE.key, IndexType.INMEMORY.toString) > then the file index is empty so we need to get the files from glob paths -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [MINOR][Testing] Test hudi-cli module [hudi]
hudi-bot commented on PR #10060: URL: https://github.com/apache/hudi/pull/10060#issuecomment-1806479127 ## CI report: * f70767fb0cf87e383474adc127779f6d31990742 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20838) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR] disable reader for test with enum [hudi]
hudi-bot commented on PR #10061: URL: https://github.com/apache/hudi/pull/10061#issuecomment-1806479164 ## CI report: * 1b2a9288f4d10a0c68336eaf97770b165a3bc773 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR][Testing] Test hudi-cli module [hudi]
hudi-bot commented on PR #10060: URL: https://github.com/apache/hudi/pull/10060#issuecomment-1806468044 ## CI report: * f70767fb0cf87e383474adc127779f6d31990742 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7076] Turn on new features by default through configs for 1.0.0-beta1 [hudi]
hudi-bot commented on PR #9998: URL: https://github.com/apache/hudi/pull/9998#issuecomment-1806467419 ## CI report: * d1f415b1400e0ff853e38b48fdaa7a9fe7aa44d3 UNKNOWN * 3b32f34ca7b86153850a6dea50b3de2c488b1b61 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20825) * 647c783c527a1b18e47feaee3fb7335b37fe5f26 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20834) * 4cabc36a8ca585b4b1f4ff845145618a70f67174 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20836) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR][TESTING] Enable configs testing 3 [hudi]
hudi-bot commented on PR #10055: URL: https://github.com/apache/hudi/pull/10055#issuecomment-1806467914 ## CI report: * 3fff661e4b61f2eec4148b3eb93110c8cdc8c8e1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20833) * f70767fb0cf87e383474adc127779f6d31990742 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20837) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [MINOR] disable reader for test with enum [hudi]
jonvex opened a new pull request, #10061: URL: https://github.com/apache/hudi/pull/10061 ### Change Logs _Describe context and summary for this change. Highlight if any code was copied._ ### Impact _Describe any public API or user-facing feature change or any performance impact._ ### Risk level (write none, low medium or high below) _If medium or high, explain what verification was done to mitigate the risks._ ### Documentation Update _Describe any necessary documentation update if there is any new feature, config, or user-facing change_ - _The config description must be updated if new configs are added or the default value of the configs are changed_ - _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make changes to the website._ ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Change Logs and Impact were stated clearly - [ ] Adequate tests were added if applicable - [ ] CI passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR][TESTING] Enable configs testing 3 [hudi]
hudi-bot commented on PR #10055: URL: https://github.com/apache/hudi/pull/10055#issuecomment-1806460194 ## CI report: * 3fff661e4b61f2eec4148b3eb93110c8cdc8c8e1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20833) * f70767fb0cf87e383474adc127779f6d31990742 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7076] Turn on new features by default through configs for 1.0.0-beta1 [hudi]
hudi-bot commented on PR #9998: URL: https://github.com/apache/hudi/pull/9998#issuecomment-1806459953 ## CI report: * d1f415b1400e0ff853e38b48fdaa7a9fe7aa44d3 UNKNOWN * 3b32f34ca7b86153850a6dea50b3de2c488b1b61 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20825) * 647c783c527a1b18e47feaee3fb7335b37fe5f26 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20834) * 4cabc36a8ca585b4b1f4ff845145618a70f67174 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HUDI-7081) Hoodie FilegroupReader cannot read Enums from MOR avro log blocks
Jonathan Vexler created HUDI-7081:
-
Summary: Hoodie FilegroupReader cannot read Enums from MOR avro
log blocks
Key: HUDI-7081
URL: https://issues.apache.org/jira/browse/HUDI-7081
Project: Apache Hudi
Issue Type: Bug
Reporter: Jonathan Vexler
{code:java}
buildFileGroupIterator(
Option.empty[PartitionedFile => Iterator[InternalRow]],
partitionValues,
Option.empty[HoodieBaseFile],
getLogFilesFromSlice(fileSlice),
requiredSchemaWithMandatory,
outputSchema,
partitionSchema,
broadcastedHadoopConf.value.value,
-1,
-1,
shouldUseRecordPosition
) {code}
This only uses StructType schemas and converts to avro. So it thinks enums are
strings
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[PR] [MINOR][Testing] Test hudi-cli module [hudi]
yihua opened a new pull request, #10060: URL: https://github.com/apache/hudi/pull/10060 ### Change Logs As above ### Impact none ### Risk level none ### Documentation Update N/A ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Change Logs and Impact were stated clearly - [ ] Adequate tests were added if applicable - [ ] CI passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR][TESTING] Enable configs testing 3 [hudi]
hudi-bot commented on PR #10055: URL: https://github.com/apache/hudi/pull/10055#issuecomment-1806397370 ## CI report: * 3fff661e4b61f2eec4148b3eb93110c8cdc8c8e1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20833) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(hudi) branch master updated (a91bdece2c1 -> 256bbe9d455)
This is an automated email from the ASF dual-hosted git repository. yihua pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/hudi.git from a91bdece2c1 [HUDI-7079] Disable new file format for metadata table (#10058) add 256bbe9d455 [MINOR] Disable reader in schema evolution tests with Streamer (#10059) No new revisions were added by this update. Summary of changes: .../TestHoodieDeltaStreamerSchemaEvolutionBase.java | 13 + .../TestHoodieDeltaStreamerSchemaEvolutionExtensive.java| 6 +++--- .../TestHoodieDeltaStreamerSchemaEvolutionQuick.java| 2 +- 3 files changed, 17 insertions(+), 4 deletions(-)
Re: [PR] [MINOR] disable reader in schema evo test [hudi]
yihua merged PR #10059: URL: https://github.com/apache/hudi/pull/10059 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [MINOR] disable reader in schema evo test [hudi]
jonvex opened a new pull request, #10059: URL: https://github.com/apache/hudi/pull/10059 ### Change Logs disable new reader explicitly ### Impact not supported currently so will pass even if reader is enabled by default ### Risk level (write none, low medium or high below) none ### Documentation Update N/A ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Change Logs and Impact were stated clearly - [ ] Adequate tests were added if applicable - [ ] CI passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR][TESTING] Enable configs testing 3 [hudi]
hudi-bot commented on PR #10055: URL: https://github.com/apache/hudi/pull/10055#issuecomment-1806338086 ## CI report: * 6dc1594a35d786c6efbf944f36c41b6e51e4acf1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20830) * 3fff661e4b61f2eec4148b3eb93110c8cdc8c8e1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20833) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7076] Turn on new features by default through configs for 1.0.0-beta1 [hudi]
hudi-bot commented on PR #9998: URL: https://github.com/apache/hudi/pull/9998#issuecomment-1806337863 ## CI report: * d1f415b1400e0ff853e38b48fdaa7a9fe7aa44d3 UNKNOWN * 3b32f34ca7b86153850a6dea50b3de2c488b1b61 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20825) * 647c783c527a1b18e47feaee3fb7335b37fe5f26 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20834) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7076] Turn on new features by default through configs for 1.0.0-beta1 [hudi]
hudi-bot commented on PR #9998: URL: https://github.com/apache/hudi/pull/9998#issuecomment-1806329969 ## CI report: * d1f415b1400e0ff853e38b48fdaa7a9fe7aa44d3 UNKNOWN * 3b32f34ca7b86153850a6dea50b3de2c488b1b61 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20825) * 647c783c527a1b18e47feaee3fb7335b37fe5f26 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-7080) Re-enable a few tests with new Spark file format and file group reader
[ https://issues.apache.org/jira/browse/HUDI-7080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-7080: Fix Version/s: 1.0.0 > Re-enable a few tests with new Spark file format and file group reader > -- > > Key: HUDI-7080 > URL: https://issues.apache.org/jira/browse/HUDI-7080 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Ethan Guo >Priority: Major > Fix For: 1.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HUDI-7080) Re-enable a few tests with new Spark file format and file group reader
Ethan Guo created HUDI-7080: --- Summary: Re-enable a few tests with new Spark file format and file group reader Key: HUDI-7080 URL: https://issues.apache.org/jira/browse/HUDI-7080 Project: Apache Hudi Issue Type: Improvement Reporter: Ethan Guo -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [MINOR][TESTING] Enable new configs by default for 1.0.0-beta - 2 [hudi]
hudi-bot commented on PR #10054: URL: https://github.com/apache/hudi/pull/10054#issuecomment-1806257277 ## CI report: * 8f2e512d3591980d75d06bfb1086ac854b12745c UNKNOWN * 321ee11654d9e2627e6677179b3c9c26e233c538 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20829) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR][TESTING] Enable configs testing 3 [hudi]
hudi-bot commented on PR #10055: URL: https://github.com/apache/hudi/pull/10055#issuecomment-1806239544 ## CI report: * 021fbdb5dfe7b04a087815adf6afa51a27686523 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20826) * 6dc1594a35d786c6efbf944f36c41b6e51e4acf1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20830) * 3fff661e4b61f2eec4148b3eb93110c8cdc8c8e1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20833) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR][TESTING] Enable configs testing 3 [hudi]
hudi-bot commented on PR #10055: URL: https://github.com/apache/hudi/pull/10055#issuecomment-1806231902 ## CI report: * 021fbdb5dfe7b04a087815adf6afa51a27686523 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20826) * 6dc1594a35d786c6efbf944f36c41b6e51e4acf1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20830) * 3fff661e4b61f2eec4148b3eb93110c8cdc8c8e1 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7076] Turn on new features by default through configs for 1.0.0-beta1 [hudi]
hudi-bot commented on PR #9998: URL: https://github.com/apache/hudi/pull/9998#issuecomment-1806231530 ## CI report: * d1f415b1400e0ff853e38b48fdaa7a9fe7aa44d3 UNKNOWN * 3b32f34ca7b86153850a6dea50b3de2c488b1b61 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20825) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7074] Flink incremental query for non-blocking concurrency control [hudi]
hudi-bot commented on PR #10053: URL: https://github.com/apache/hudi/pull/10053#issuecomment-1806209456 ## CI report: * 240ab68dc60b1d30983a6b1c7a497e9d5b4d4e16 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20822) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7079] disable new file format for metadata table [hudi]
yihua merged PR #10058: URL: https://github.com/apache/hudi/pull/10058 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
(hudi) branch master updated: [HUDI-7079] Disable new file format for metadata table (#10058)
This is an automated email from the ASF dual-hosted git repository.
yihua pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/master by this push:
new a91bdece2c1 [HUDI-7079] Disable new file format for metadata table
(#10058)
a91bdece2c1 is described below
commit a91bdece2c1782410009915881ff6488fdfbf973
Author: Jon Vexler
AuthorDate: Fri Nov 10 13:04:42 2023 -0500
[HUDI-7079] Disable new file format for metadata table (#10058)
Disable the new file format for the metadata table because we haven't
implemented hfile base files yet. Also made new method for duplicated code
Co-authored-by: Jonathan Vexler <=>
---
.../java/org/apache/hudi/client/BaseHoodieTableServiceClient.java| 2 +-
.../src/main/java/org/apache/hudi/table/HoodieTable.java | 2 +-
.../java/org/apache/hudi/table/action/compact/CompactHelpers.java| 2 +-
.../org/apache/hudi/common/bootstrap/index/HFileBootstrapIndex.java | 4 +---
.../java/org/apache/hudi/common/table/HoodieTableMetaClient.java | 5 +
.../src/main/scala/org/apache/hudi/DefaultSource.scala | 4 ++--
6 files changed, 11 insertions(+), 8 deletions(-)
diff --git
a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieTableServiceClient.java
b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieTableServiceClient.java
index 8b39a46cb07..112b8111004 100644
---
a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieTableServiceClient.java
+++
b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieTableServiceClient.java
@@ -952,7 +952,7 @@ public abstract class BaseHoodieTableServiceClient
extends BaseHoodieCl
// from the async indexer (`HoodieIndexer`).
// TODO(HUDI-5733): This should be cleaned up once the proper fix of
rollbacks in the
// metadata table is landed.
- if (isMetadataTable(metaClient.getBasePathV2().toString())) {
+ if (metaClient.isMetadataTable()) {
return
inflightInstantsStream.map(HoodieInstant::getTimestamp).filter(entry -> {
if (curInstantTime.isPresent()) {
return !entry.equals(curInstantTime.get());
diff --git
a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java
b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java
index c44d3b0f4cb..aa521fcc3e5 100644
---
a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java
+++
b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java
@@ -1010,7 +1010,7 @@ public abstract class HoodieTable implements
Serializable {
// Only execute metadata table deletion when all the following conditions
are met
// (1) This is data table
// (2) Metadata table is disabled in HoodieWriteConfig for the writer
-return !HoodieTableMetadata.isMetadataTable(metaClient.getBasePath())
+return !metaClient.isMetadataTable()
&& !config.isMetadataTableEnabled();
}
diff --git
a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/CompactHelpers.java
b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/CompactHelpers.java
index 6a1e3f14062..78ea56fd9ad 100644
---
a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/CompactHelpers.java
+++
b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/CompactHelpers.java
@@ -106,7 +106,7 @@ public class CompactHelpers {
}
public Option getInstantRange(HoodieTableMetaClient
metaClient) {
-return
HoodieTableMetadata.isMetadataTable(metaClient.getBasePathV2().toString())
+return metaClient.isMetadataTable()
? Option.of(getMetadataLogReaderInstantRange(metaClient)) :
Option.empty();
}
diff --git
a/hudi-common/src/main/java/org/apache/hudi/common/bootstrap/index/HFileBootstrapIndex.java
b/hudi-common/src/main/java/org/apache/hudi/common/bootstrap/index/HFileBootstrapIndex.java
index 27314f150dc..744f3dc3a2d 100644
---
a/hudi-common/src/main/java/org/apache/hudi/common/bootstrap/index/HFileBootstrapIndex.java
+++
b/hudi-common/src/main/java/org/apache/hudi/common/bootstrap/index/HFileBootstrapIndex.java
@@ -34,7 +34,6 @@ import org.apache.hudi.common.util.collection.Pair;
import org.apache.hudi.exception.HoodieException;
import org.apache.hudi.exception.HoodieIOException;
import org.apache.hudi.io.storage.HoodieHFileUtils;
-import org.apache.hudi.metadata.HoodieTableMetadata;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
@@ -106,8 +105,7 @@ public class HFileBootstrapIndex extends BootstrapIndex {
FileSystem fs = metaClient.getFs();
// The metadata table is never bootstrapped, so the bootstrap index
Re: [PR] [HUDI-7079] disable new file format for metadata table [hudi]
hudi-bot commented on PR #10058: URL: https://github.com/apache/hudi/pull/10058#issuecomment-1806157600 ## CI report: * 2227bae3742151691a638a70c94c9d44118be13a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20831) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR][TESTING] Enable configs testing 3 [hudi]
hudi-bot commented on PR #10055: URL: https://github.com/apache/hudi/pull/10055#issuecomment-1806157555 ## CI report: * 021fbdb5dfe7b04a087815adf6afa51a27686523 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20826) * 6dc1594a35d786c6efbf944f36c41b6e51e4acf1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20830) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR][TESTING] Enable new configs by default for 1.0.0-beta - 2 [hudi]
hudi-bot commented on PR #10054: URL: https://github.com/apache/hudi/pull/10054#issuecomment-1806157511 ## CI report: * 8f2e512d3591980d75d06bfb1086ac854b12745c UNKNOWN * 7be659422f59706308364bdab2ce3e78a2f08812 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20823) * 321ee11654d9e2627e6677179b3c9c26e233c538 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20829) Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [HUDI-7079] disable new file format for metadata table [hudi]
hudi-bot commented on PR #10058: URL: https://github.com/apache/hudi/pull/10058#issuecomment-1806147191 ## CI report: * 2227bae3742151691a638a70c94c9d44118be13a UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR][TESTING] Enable new configs by default for 1.0.0-beta - 2 [hudi]
hudi-bot commented on PR #10054: URL: https://github.com/apache/hudi/pull/10054#issuecomment-1806147053 ## CI report: * 8f2e512d3591980d75d06bfb1086ac854b12745c UNKNOWN * 7be659422f59706308364bdab2ce3e78a2f08812 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20823) * 321ee11654d9e2627e6677179b3c9c26e233c538 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [MINOR][TESTING] Enable configs testing 3 [hudi]
hudi-bot commented on PR #10055: URL: https://github.com/apache/hudi/pull/10055#issuecomment-1806147122 ## CI report: * 021fbdb5dfe7b04a087815adf6afa51a27686523 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20826) * 6dc1594a35d786c6efbf944f36c41b6e51e4acf1 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (HUDI-7079) Disable new file reader for metadata table
[ https://issues.apache.org/jira/browse/HUDI-7079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-7079: - Labels: pull-request-available (was: ) > Disable new file reader for metadata table > -- > > Key: HUDI-7079 > URL: https://issues.apache.org/jira/browse/HUDI-7079 > Project: Apache Hudi > Issue Type: Improvement >Reporter: Jonathan Vexler >Assignee: Jonathan Vexler >Priority: Major > Labels: pull-request-available > > hfile base files are not yet implemented, so the metadata table should use > the hoodie relations -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [HUDI-7079] disable new file format for metadata table [hudi]
jonvex opened a new pull request, #10058: URL: https://github.com/apache/hudi/pull/10058 ### Change Logs Disable the new file format for the metadata table because we haven't implemented hfile base files yet. Also made new method for duplicated code ### Impact metadata table will work ### Risk level (write none, low medium or high below) low ### Documentation Update N/A ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Change Logs and Impact were stated clearly - [ ] Adequate tests were added if applicable - [ ] CI passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (HUDI-7079) Disable new file reader for metadata table
Jonathan Vexler created HUDI-7079: - Summary: Disable new file reader for metadata table Key: HUDI-7079 URL: https://issues.apache.org/jira/browse/HUDI-7079 Project: Apache Hudi Issue Type: Improvement Reporter: Jonathan Vexler Assignee: Jonathan Vexler hfile base files are not yet implemented, so the metadata table should use the hoodie relations -- This message was sent by Atlassian Jira (v8.20.10#820010)
